<!DOCTYPE SGMLguid>
<SET TAG=H1 ITEM=BODY VALUE=TOP>
<SET TAG=P ITEM=SK VALUE=0>
<SET TAG=XMP ITEM=IN VALUE=0>
<SET TAG=XMP ITEM=FONT VALUE=MONO>
<SET TAG=P ITEM=IN VALUE=0>
<SET TAG=UL ITEM=SK VALUE=0>
<SET TAG=H1 ITEM=FONT VALUE=TITLE>
<SET TAG=H2 ITEM=FONT VALUE=HEAD>
<SET TAG=H3 ITEM=FONT VALUE=HEAD>
<SET TAG=H4 ITEM=FONT VALUE=|roman10|>
<GDOC HY=USER>
..ti set
..tb 9 17 25 33 41 49 57 65 73
<FRONTM>
<TITLEP>
<TITLE STITLE="RPC Internals">
<TITLE>RPC Internals

..ce (Formerly, "RPC Implementation Guide")


<AUTHOR>T.J. Berners&hyphen.Lee   CERN DD/OC

 
<DATE>Version 2.3.2,  Last Revised February 1989
<COPYRIGHT>CERN 1986, 87, 88, 89

..fo on
This document provides internal details of the RPC system which are not covered
by the RPC User Manual, and which a user should not need to know.
In particular, it defines internal interfaces within the RPC system
and message formats.
..fo off


..ce on
<HP1>(Details about how to acquire a
copy of this document are given in an appendix).

DOCFIND RPC INTERNALS RPCIMP
</HP1>
</TITLEP>
<TOC>
</FRONTM>
<BODY>
<H1>Introduction
..tb 9 17 25 33 41 49 57 65 73
..tb set        
..hy set endpt 3
..hy set minpt 3
..ti set

This document was written for those who intend to
maintain, modify or port the RPC system, or (to a lesser extent)
those who have an academic interest in its internal design.
It assumes a prior knowledge of RPC principles, and of the
User Manual <BIBREF REFID=RPCUSER> for this system.
The amount of overlap with that manual has been reduced as far as
possible, but has been unavoidable in some places.

This document covers the internal functioning of the RPC system itself.
In particular, it defines internal interfaces within the RPC system
and message formats.
It does not cover code which was developed in order to make
some of the target environments more friendly, such as the
AST system or the Event Handling system under MoniCa.

There is some discussion of design criteria and choices,
with comparison with other systems.

The level of detail covered has had to be a compromise, in an effort
to answer a reasonable number of likely questions in a reasonable
space and time.
<H1>Design Choices
<i1 ix=1>Design choices
This section introduces this particular RPC
system to those who are familiar with other RPC systems,
explaining some of the eccentricities of this system.
These have followed from the requirements of the system,
in particular the environments in which it has had to run.
<h2>Requirements and Constraints
<i1 ix=1>Constraints, on design
<h3>Target environments
RPC is used
for a variety of applications
in control and data acquisition systems for physics experiments
and related test equipment.
These systems must be distributed for three reasons:
the severe geographical constraints,
the system topology required to handle very large data rates,
and the fact that the design teams are distributed between
collaborating departments.

In our case, processors involved are
VAX
..bf FOOT;
&S'TM.
..pf
minicomputers, and 32 bit (M68000 series) and 16 bit (M6809)
microprocessors, and personal computers (IBM and Macintosh).
The operating environments are varied, and Unix is
(at the time of writing) conspicuous by its absence from
the online systems.
Some of the systems are monotask systems, or (in VMS)
systems in which process creation entails a large overhead.
This lead to te use in many cases of "AST" (see Glossary)
procedures, rather than the frequent forking of processes.
<h3>Example applications
Examples of the use of RPC include facilities such as the following,
which have been provided remotely using RPC by various groups
at CERN at elsewhere:
<ul>
<li>Remote error logging
<li>File access
<li>Terminal access
<li>Graphics packages
<li>Menu packages
<li>Gas system control and monitoring
<li>Control of processor farms
<li>Remote program load and control
<li>Remote database access
<li>Remote FASTBUS access
<li>Remote operating system commands
<li>etc.
</ul>
<h3>Communications protocols
<i1 ix=1>Protocols, requirements
Whereas many commercial and academic RPC systems have been
based on the Internet protocols,
these we not generally available in most of the online
environments.
<ul>
<li>DECNET
..bf FOOT;
&S'TM.,
..pf
<li>raw Ethernet
<li>ISO Class 4 Transport,
<li>direct communication over 32 bit parallel cables,
<li>RS232 links
</ul>
are currently used, with the adition of others
possible in the future.
The system had therefore to be designed to cope with both
unreliable datagram and reliable connection oriented services,
and to allow a migration path for users as the availability of
facilities changed.
It is general required to be able to reconfigure an application
to run over a different protocol at run time,
by changing tables or by control by the program itself.

The use of unreliable datagram services was considered to be
a temporary phase, and was covered by the invention of
an ad hoc protocol summarised later in this document.
<h3>Application Languages
As FORTRAN has been a favorite language for physicists for a long time,
there was a requirement to provide RPC stubs to interface to
FORTRAN code.
The first RPC application (The Valet-Plus) involved a lot
of Pascal code, so Pascal compatibility was also required.
Later, the OS9 system became important for users, and only supported
C, and the Valet plus environment produced a requirement for stubs
in PILS.

In some cases (when using the CERN cross software, on the
PCs, and originally on VAX/VMS)
the support for Pascal was, when the system was developed,
much stronger for the support for C.
Support for FORTRAN was patchy in the microprocessor and PC areas,
especially if it to be mixed with other languages.
<h2>Design choices
<h3>Implementation Language
For the RPC system itself, FORTRAN was rejected as having insufficient
power (no pointers or structures) and being insufficiently available.
Because of its availablity at the time,
Pascal was chosen as the initial implementation language
for the system, whereas C would have been more usual.
Since that time, support for C oriented environments has
been provided, using the help of pascal to C conversion programs.

FORTRAN callable stubs were provided for
some time by producing "FORTRAN callable"
pascal, but eventually it proved simpler to provide stubs in FORTRAN
itself, and provide FORTRAN-callable entry points in the run time
support.
<i1 ix=1>Languages, implemention
(The support for N languages on M machines tends to produce N*M
mappings between them, e.g. between FORTRAN and C.
It is easier to support the
mapping in the limited case of run-time library entry points,
rether than in the general case of a stub procedure produced by the
RPC compiler. One then has only to support N compiler options
and M runtime library flavours).

The same became true of C and PILS
<i1 ix=1>PILS
<REFER ID=PILS>, so the RPC compiler now produces four different languages.

<h3>Description Language
..cc 6
In our case, we chose an Ada
..bf FOOT;
&S'TM.
..pf
-like language to describe
the procedure interfaces,
as it was the only real time language in use locally which
provides sufficient information about each parameter, namely
its type and direction.
For none of the other programming languages used was this true.
<h3>Message format

Standardisation of the
format used in RPC messages is still an open question,
despite its obvious advantages for interworking.
One of the reasons for this is that different RPC systems
have been optimised according to
different, conflicting, criteria.
There is a trade&hyphen.off, for example, in the coding scheme
between generality, economy of message length, and encoding speed.

In our case, we chose the Xerox Courier
format, in which data sizes are defined at compilation time,
and data is marshalled into bytes
most significant byte first.
The overhead of encoding data according to the X.409 standard
was felt to be too high, both in time and space.
In particular, transmission over RS232 lines put stringent constraints
on the efficiency of encoding.
Some systems deliberately check whether the
communicating machines happen to be of the same type,
and in this case skip the conversion of data into standard form.
The time, space and complexity of this was not
felt to be justified by
the relatively small amount of time
spent converting data formats.
(This is especially so when the conversion routines called by the
stub code are expanded in&hyphen.line by a good optimising compiler
such as the VAX/Pascal compiler)

<h3>Data Types
The primary requirement was for FORTRAN style data types,
but the type offered were extended on demand to
the full set of types used by Pascal and C,
with the exception of variant records (unions).

We had to extend the Courier data types to include
a real number type (sent in IEEE format).
We also found that user program portability is greatly enhanced by the addition
of a type which is represented as the native integer on any
machine, but is constrained to lie in a 16 bit range, and is passed as
16 bits.
The string type proved to have the greatest variation in its
local representation, each version of pascal having its own representation,
and a semantics slightly different from that of FORTRAN.

In general, it was found to be important that the logical type
of the data transferred be specified quite independently of the
actual local format.
The first determines the format of the messages,
while the second determines the method by which the data is
passed on the stack.
The mapping between these two is a function
of options supplied to the rpc compiler when the stubs are generated.
It is not, for example, so useful to have a data type "zero terminated
string": the data type is "string", which is mapped onto
a zero&hyphen.terminated string
when stubs are generated in C.
The same procedure, written in FORTRAN, will use a string descriptor,
while in pascal it would use one of the more or less exotic
devices built into various compilers, or,
for standard pascal, a pair of separate variables
for the characters and the length.

<h3>Addressing and Identifiers
Unlike in some systems, no numbers or identifiers
need to be globally unique.

<i1>Unique identifiers
The procedure number within the package is only known by the
client and server stub modules, and is allocated at compilation time.
The other three layers are allocated at load time.

The package number is generated by,
and only relevant in the context of, the server on which it it provided.
It could be used as a capability in a capability&hyphen.based
authorisation system.
The client stub module knows only a logical name for its server counterpart:
this is used to look up the full address
of the remote package.
Although this process currently uses local tables which are
set up when the system
is configured, provision is made for the search to be extended to
a heirarchical sequence of name servers.

We find that a distributed system is rarely built as a monolithic
whole,
but grows by the progressive amalgamation of
independently designed sub systems.
It is therefore important that a system may function independently
as much as possible, while also being part of a larger system.
The principle of information hiding in modular software then
applies equally to distributed software,
and requires that one does not rely on globally unique
identifiers or centralised administration.
 
<h3>Call Time vs. Load Time

<i1>Call time
<i1>Load time
Trade&hyphen.offs exist when designing the run&hyphen.time support
software for RPC.
There is a distinction between systems designed for
an occasional isolated call,
and those designed to handle large amounts of traffic between
particular partners.
In our case, the goal was to optimise
run&hyphen.time performance by
doing as much work as possible at compilation time (type checking, for
instance), and at program load time. 

At load time, any necessary connections are established between tasks. The
media to be used, and the remote addresses, are looked up at this time. This
allows a system to be reconfigured without recompilation or even
relinking the executable images.
Load time is also the appropriate time for starting server tasks, and checking
client authorisation.

Under the VAX/VMS
..bf FOOT;
&S'TM.
..pf
operating system, PILS, and TurboPascal, facilities exist
for the automatic intialisation
of software modules when they are loaded.
In this case,
any stub modules declare themselves to the run&hyphen.time support
at load time.
The remote binding is then carried out quite
invisibly to the programmer, who does not have to write any code
<i1>Initialisation, automatic
to do it himself.

Although it is possible for a program to expressly relink itself to different
modules while it is running, in general any remote call is made over
an established communication path to an established server module, and
can therefore execute with no additional delay.
This method of working also allows us to use connection&hyphen.oriented
communications services
(which happen to be available locally and are at a more advanced
stage of standardisation than connectionless services)
without a prohibitive overhead.

The call&hyphen.time
performance is then determined by parameter copying time and by
operating system overheads. Without any assembly coding in
these areas, a dummy call with n bytes of user data takes, across Ethernet,
for example, about (5+0.08n) ms between VAX and an M68000
based system.
<h2>Extra Facilities
Although the primary goal was simply transparent procedure call
between machines,
experience has lead to a number of extra features, beyond the basic RPC
function.
These include
<ul>
<li>detailed trace outputs about remote calls and their
parameters, to help in the debugging of distributed applications.
<li>
When the confidence in the remote machine, the remote application software,
or the communication protocol is low, users have asked for
optional application level
timeouts on remote operations.
This turns out to be important in a real time system for preventing
some unforseen fault causing an unduly large amount of the system to hang.
In this case, a maximum time allowed for a remote operation
may be specified as a compiler option, or in the definition file.
See PRAGMA TIMEOUT in the user manual.
<i1>Pragmas
<i1>Timeouts
<li>Some users (especially FORTRAN programmers) have variables
whose actual data type vairies at run time.
There is provision for users to include their own marshalling algorithms
to cope with this (on the assumption that <hp2>some</hp2>
algorithm exists for interpreting the data).
This also allows related implicit data such as FORTRAN common blocks
to be transferred if necessary.
<i1>COMMON blocks, transfer of
See PRAGMA EXTERNAL_MARSHALLING in the user manual.
<li>
Also selectable at compilation time is the option of remote
operations, once started, continuing concurrently with the calling program.
This only applies to procedures which return no data.
By adding a little pipelining, this option increases the performance
of some control applications.
See PRAGMA concurrent in the user manual.
<li>Multiple servers: Dynamic Binding.
In some cases, a client may have the choice, at run&hyphen.time,
of many different, though congruent, server modules.
For example, a control program may have access to identical servers on
different parts of the apparatus.
For these applications, facilities are provided
for a client program to reconfigure itself to use different server modules,
and to switch the binding rapidly between them.
Obviously, in this case, the application program does become aware of
the addresses of server modules, so the transparency is reduced.
<li>Asynchronous servers.
The lack of multitasking in many environments has produced
a requirements for a server which will run application code, but then
be interrupted at AST (see Glossary) level to service a remotely called procedure.
<li>Two forms of reverse call proved to be required.
In one ("callback"), a call is made from the server process back to the
original client process during the original call.
In the other ("CALLER addressing"), the server merely establishes
a handle on the caller
in order to call him back later asynchronously,
outside the bounds of the original call.
</ul>
<h2>Retrospective comments
In considering the design in the light of experience,
there are a few areas in which things might usefully have
been done differently.
<ol>
<li>The use of Courier format has been useful only in
that it answered some basically arbitrary questions, and was stable.
It has not, in practice, been used to interwork with any other systems
to date.
If one were to change the format, then
from the technical point of view, the version number system
<fn>
The ideal version numbering would involve the specification of
three numbers each particular version of the definition file.
One would be the version number (V) of the file; one would be the
lowest previous version number of server with which a client of version V
would be compatible, and the other the lowest (previous) version number
of client with which a server of version V would be compatible.
The checking of version numbers would involve sending two of these
from one side to the other.
</fn>
could have been more sophisticated, and checked at initialisation time,
rather than every call.
Also, the TID field of the message belongs to a lower layer of protocol,
and so should have been included in an outer layer of message format.
<li>The limitation of fixed size buffers has been awkward in some
cases. It would have been useful to be able to send arbitrary large
messages.
There is a question as to whether this would compromise the speed.
This is discussed later in more detail, with some workarounds.
<li>If we had realised how long the ad hoc datagram protocol would be used,
we would perhaps have made it more sophisticated.
The deficiencies of this protocol are also discussed later.
</ol>
A list of possible extensions and improvements to the current system
is kept elsewhere, as it changes (in both directions) from time to time.
<H1>Operational Overview
There follow a few notes on the way a remote call works.
Some of this information is duplicated in the
user manual.
For more detail, see the flowcharts
of the procedures in question, and
comments in the source code.
..cc on
..fo off
..bx 5 35 / 45 75
..tb set $
..tb 20c 40c 60c
$Client application$User$Server application
$module$package$module
$$calls
..bx 20 / 60
..bx new 40
..bx off
..bx off
$User package$defined by$User package
$calls$- definition file -$calls
..bx 20 / 60
..bx new 10 30 / 50 70
$Client stub$Courier$Server stub
$$Level 1
..bx off

..bx can
$RPC_xxx$$RPC_xxx
$calls$$calls
..bx on 20 / 60

..bx new 5 35 / 45 75
$RunTime Support$Courier$RunTime Support
$(portable)$$(portable)
..bx off
..sk
..bx can
$TS_xxx$$TS_xxx
$calls$calls
..bx on 20 / 60
..sk
..bx new 5 35 / 45 75
..sk
$General Communications$$General Communications
$Support$$Support
..bx off
..sk
..bx 5 35 / 45 75
..br
$Specific underlying$(whatever$$Specific underlying
$medium$protocol)$medium
..bx off
..fo on
..ce on
..us on
..tb set

Software interfaces in an RPC link.
<i1 ix=1>Interfaces, diagram of software
Vertical interfaces are procedure calls between layers of software;
horizontal interfaces are protocols at each level.
..us off
..ce off
..cc off
..tb 9 17 25 33 41 49 57 65 73
<H2>RPC_Open: setting up an RPC link

<i1>RPC_Open
The addressing and routing decisions which are necessary in order to connect
two modules are made, as far as is possible, at initialisation time,
in order to allow remote calls to run with as little overhead as possible.
This has been referred to elsewhere
<BIBREF refid=ecma127>
as "semi&hyphen.lazy evaluation of binding".

<i1>Binding
A procedure is provided to set up an RPC service.
It is called by, or on behalf of, the client stub module in order to
establish contact with its partner server stub.
The package required is identified by
a name (a string of characters). The procedure, RPC_Open, is part
of the RPC run&hyphen.time support. RPC_Open then returns to the caller a
<i1>Handle
<hp2>handle</hp2> (an integer) to be used when referencing the package. 

When RPC_Open is called, it in fact sets up a table within the local RPC RTS
which gives, for that "handle", the network address of the relevant remote
<i1>handle
server, and the package number which will identify the package on the remote machine.
The caller is unaware of these values. 

Under VMS, the logical name table is currently used for name translation, and
under MoniCa, the MoniCa symbol table.  Both allow interactive setting up using
<i1>DEFINE command
a DEFINE command. See the user manual for details for other systems.

(The translation, or mapping, function is related to the verification
of the user's right to access the remote package.
(authorisation).
Some work has been done on this, including a definition of requirements
and a prototype database access implementation.
This work has not yet been integrated with the RPC system.)
..bx off

The client stub module xxx contains a procedure open_xxx which, when called,
will call RPC_Open, with its own logical name, and hence acquire a handle
which it can use for future calls.
Under VMS, this procedure is flagged as an "initialise" procedure,
and called automatically on image load.
It is also called automatically under TurboPascal
on the IBM PC or the Macintosh, or in PILS.

<H2>Using the link: Stub Calls Run&hyphen.Time Support

Having packed the buffer, the stub calls the run&hyphen.time support procedure
RPC_CALL.
It passes across the package handle, and the buffer.

<H2>RTS calls Communication Service

<i1>RPC_Call
The RPC_CALL procedure performs a little more formatting of the message,
inserting the reference number of the package on the remote machine. The
run&hyphen.time support package uses the package handle to extract this from its
tables, and also to get details of the how to contact the remote server.  This
includes which network to use, if there is more than one (eg RS232 and Ethernet),
and any addresses which it will need to use on that network. If the
communication service it uses is connection&hyphen.oritented, then it will use
a connection which has already been set up by RPC_Open. 

Standard calling sequences in this area have now been defined by the
CATS committee.
The TS_xxx routines which it proposed are similar to the
TS_xxx routines currently used by this system, and documented in a
later section.

<H2>Adaptation of communications calling sequence

<i1>Transport service, adapter layer
A layer of software exists which adapts the standard calling sequence
and functionality required by the RPCRTS to the local facilities
available is written specially as part of the port to each system.
This layer provides four functions:
<ul>
<li>Selection of the apropriate medium
<li>
Calling the local routines, OS entry points, etc,
to communicate over that medium
<li>
Handling of multiple simultaneous connections, for connection oriented
services.
<li>Protocols for ensuring reliability, over unrealiable datagram
communication services.
</ul>
See "Transport Service" index entries.
<H2>Server RTS system calls communication service.

On the remote ("server") machine, the Run&hyphen.Time Support system is
waiting for a message.

The calling seqeunce here is again dependent on the communication service used.
Depending on the type of server, blocking or asynchronous receive calls are
used. 

<i1>spawning of servers
The question of whether the RTS process forks (spawns) off a separate process
to handle the call is up to the RTS in question.
This is not currently an option.
Under VMS, the overhead of process creation is very great.
Under monotask systems (PC etc) it is impossible.
Under the TCS system (DD/OC MODEL project VMS subtasks),
a fork is made for each request. (This has been built up using the
RPC_Queue_Server" and RPC_Service" primitives.)

<H2>Server RTS system calls Server Stub

When a message is received, the server RTS calls the server stub.

<i1>Server stub, overview
There is one stub procedure for each "package".
Inside it is a big CASE statement, ("computed GOTO") to select, on the
procedure number, the relevant unpacking/packing code and user procedure.

<H2>Server stub calls User routine

The User procedure is called in just the same way as it would have been called,
were the calling program on the same machine.

<H2>The Return of OUT Parameters

Before the server stub calls the user routine,
in the case of a "concurrent" procedure (not function)
with only "IN" parameters,
the stub calls RPC_early_return. This sends a reply back to the caller, who
can then continue to execute while the called routine continues
in paralel.

If an error occurs, the stub may set the m_status field in the
message to a bad (even) value, and the call will be rejected,
with the given as rejection reason.

In the normal case, when the stub returns to the RTS, the return parameters,
which have all been packed back into the message buffer, are  returned to
the caller.

<H2>Remote Errors

<i1>Error codes
Errors may occur on the called machine, either in the RTS or in the called
routine.  In either case, the errors will, where possible, be trapped,
and a "reject" response given. In this context it would be very convenient
to have a global format for error codes, so that the reason code can be
interpreted on the remote machine.

The portable RTS always generates VMS&hyphen.style error codes, with a special
facility number.

The Courier "reject" message type is used for this feature,
with the VMS error as a parameter.

<H2>Asynchronous Events, Interrupts and Break&hyphen.in

In order to allow asynchonous events to be handled remotely, the
RTS must allow a client stub so be called as a result of an unsolicited
call message arriving.  This requires that the communications service
be capable of calling the RTS assynchonously, and that a priority channel
be allocated to such messages.
This is required, for example, in order to implement remote break&hyphen.in
in a PILS or MoniCa system.
<H2>Closing a service
The party who calls RPC_Open is responsible for calling RPC_close when
the service is no longer required.
<h2>RPC buffers
<i1>Message pool
<i1>Message format
<i1 see="Message">Buffer
The RPC system uses a pool of fixed length buffers.
The buffer length is defined (by the constant MAXLENGTH in the
RPCCONST header file) when the library is built.
Currently, the standard version has a buffer length if 1500 bytes,
and the "large" version a length of 8000 bytes.
<h3>Buffer Format
<i1>Message, format in memory
<fig place=float frame=none>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip16
<PICTURE NAME=FLOW16$S>
...skip16
<figcap><hp1>The format of an RPC_Message structure in memory.
Alternative uses of the same space are separated horizointally.
The maximum size of the message depends on the version of the
run time system used.
</hp1>
</fig>
The buffer format in memory consists of three parts.
The first part is reserved for the internal use of the RPC RTS
module and TS. It contains a link (for threading onto lists),
a status longword for the general status of the
transaction, which may be overwritten at any stage.
There is also an index into the third part of the buffer which
is used as a read and write pointer by the marshalling routines,
and there is the number of the socket over which the call is being
made.

The second part is reserved for internal use by the transport service.
It may be used for formatting a header to the message, where the
communications interface cannot gather data from dispersed buffers:
this saves the overhead of copying the data into an assembly area.

The third part of the message structure is that which is actually transmitted
between the parties. It's format is basically Courier standard,
and is described later.
<h3>Buffer passing and pointer swapping
Messages are passed around by pointers, and in general the TS layer
reserves the right to switch pointers. That is, the pointer
itself is passed VAR, so that, by overwriting the pointer,
the TS system can return a different
buffer without having to copy the data.
Therefore, for instance, in an M68020 system with a LANCE interface
chip, the packet is read into a buffer by the LANCE DMA.
The buffer is swapped with the empty one provided by the RPC system's
READ request, so that
the unmarshalling  routines take it straight out of that buffer
into the user memory area, with no extra copies at all.
<h3>Buffer allocation strategy
Some systems have more useable heap systems than others.

<i1>Heap, use of
In cases where the heap does not exist,
or where unrealistic constraints are put on its use,
the buffers are taken from a
static pool of messages (array
..bf MONO
static_messages
..pf
).
This is controlled by the precompilation option NOHEAP.
<i1>NOHEAP precompilation option
At initialisation time, the messages in the array are put onto
the free list.
The size of the initial pool is selected at a compile&hyphen.time as a
function of the environment.
One must remember, in this context, that the ethernet and RS232 protocols
require an extra buffer per socket to store the last call or the last
reply sent (or both, if callback occurs).
Under MoniCa, the LANCE chip requires (currently 4) buffers as
a receive packet pool.

In the case where the heap is usable,
buffers are allocated from that.
Even in this case,
once a buffer has been acquired by the RPC system, it is
kept permanently. When not in use, it is kept on a free list
rather than being returned to the heap.
This means that during normal running, buffers can be allocated and deallocated
very rapidly, which is essential to the efficiency of the communications
code.
(Under VMS, Eileen Berman/FNAL found that a large proportion of the time
when running over RS232 was actually spent in pascal 'new'
when this was not done)
<h3>Buffer size limitation
When an application requires larger data transfers than the buffer size,
it is necessary to get around the buffer size limitation.
I note here a few techniques.
<ol>
<li>One possibility is always to write a loop in the application,
to transfer the data in chunks.
This was done (See note by Jorgen Petersen) for the HMINI
histogram retrieval routines.
<li>In the long term, it would be possible to build in segmentation
algorithms into the RPC system. Unfortunately, checking
explicitly for buffer overflow would slow down the marshalling.
In principle, though, the compiler could detect when the parameters
could be crossing buffer boundaries, and could (only in those cases)
insert code
which checks and allows segmentation.
This would allow the much more common short RPC to remain optimal.
A possible future project.
<i1>Message size limitation
<li>As buffer sizes become significantly large,
the marshalling/unmarshalling time starts to dominate the
communication overhead for a fast medium.
In the subset of cases in which one is communicating between like
machines, it then becomes very much more efficient to transfer
the data directly between user memory and user memory,
with no intermediate buffering.
The CERN Host Interface (CHI), because it can byte swap, in fact allows
direct transfer even between machines with different byte ordering.
In this case, the (hand modified) stub does an explicit transfer of the data.
This is used in the CHI for the remote FASTBUS block operations,
when the block size excedes a 1kByte.
The buffer size at which one resorts to this technique
 is a question of trade&hyphen.off
between the reduced marshalling time and the increased overhead for
one extra DMA.
The same technique could be used over DECNET between two VAXes,
if required.
</ol>
<h1>Compiler Overview

<i1>Compiler overview
<i1 see="Compiler">RPCC
The RPCC compiler
<fn> This section is in part derived from comments by Antonio Pastore in
the source code.
</fn>
is organized in two parts:
<ol>
<li>The PARSER
<li>The CODE GENERATOR
</ol>
<h2>Parser
<i1>Parser
The PARSER checks the input for SYNTACTICAL and LEXICAL errors and
produces as output two trees, the TYPE and BLOCK trees.
It uses a lexical analyser (GETOKEN) which returns tokens
(identifiers, numeric values etc) and checks for reserved keywords
of the langage. It can backtrack by one token: this is used
for error recovery.

The user defined data types are descibed by
a list of named types, each of which
has a name, and a pointer to the type descriptor tree.

A type descriptor tree has as elements a structure which
can define a simple type, or a complex type in terms of pointers
to the descriptors of its subtypes.
Field for each type also exist to define the maximum and
minimum sizes of the type, in standard representation,
and flags for any special options (external marshalling, etc.).

<i1>Block, in RPCC
A BLOCK in this context means either a procedure or a function.
The BLOCKS tree (pointed to by BLOCKPTR) is a linked list of record each
describing a block. Each BLOCK has a
NAME, (as given by the user), and a list of parameters.
Each record of this list has a NAME, the parameter's name (as given by
the user), and a TYPE associated to it. This is a pointer to a
type descriptor.
The same routine as used to build a TYPE definition is used to
analize a parameter declaration inside a BLOCK.

<h2>Code generators
The CODE GENERATOR reads the BLOCK tree and produces as output two
files, the CLIENT and the SERVER stubs.
It also optionally produces a declaration file in Pascal or C,
with or without a definition of the user's types.

Three code generator modules currently exist.
One output generator (CODEGEN) will produce Pascal or C stubs; there is also
one (CODEFOR) to produce FORTRAN, and one (CODEPILS) for PILS.

<i1>Code generators
They tend to use similar techniques, although in places the generation
of Pascal types proves a simpler task to perform recursively than
the generation of C or FORTRAN types.
<h2>Source of the compiler
The compiler is written with the (rare) use of the INCLUDE preprocessor.
This is used to select machine&hyphen.dependent parts, maily in the
OPTIONS module.
Versions for Bsd 4.2 Unix/Ultrix, for VAXVMS and for the IBM PC
can be generated in this way.
The source is divided into modules:
<dl>
<dt>MAIN
<dd>The main compiler file, which INCLUDEs each of the following modules
in the order given:
<dt>DECGLOB
<dd>The declarations of constants, data types, and variables
for the compiler as a whole.
<dt>ERROR
<dd>The error reporting and handling code.
<dt>DEBUG
<dd>The code to list the completed trees created by the compiler.
This code is invoked by the debug options. (See user manual).
<dt>INIT
<dd>General initialisation code.
This includes setting up data areas which are effectively constant,
as strict Pascal does not provide for constant data declarations.
<dt>OPTIONS
<dd>This module handles the command line options. It contains
implementations of unix style arc and argv functions for
every system under which the compiler runs.
Also, included are the (machine dependent) file open and file close procedures.
<dt>GETOKEN
<dd>The lexical analyser. GETOKEN is also the name of the principle
procedure of this module.
<dt>PARSER
<dd>The parser.
<dt>CODEFOR
<dd>The code generator for FORTRAN stubs.
<dt>CODEPILS
<dd>The code generator for stubs in PILS.
Derived from CODEFOR. PILS has more limited data types than fortran,
but better string handling.
<dt>CODEGEN
<dd>A combined code generator for pascal and C.
This module contains a certain amount of special code for different
versions of Pascal, especially for the TurboPascal versions.
TurboPascal requires UNIT declarations, and IMPLEMENTATION and
INTERFACE sections producing separately.
</dl>
<H1>Transport Service
<H2>General
<i1>Transport service overview
The Transport module (TS) is responsible for all the communications
required by the RPC system.
As normally most of the communications code is provided by existing
operating system facilities, the TS module acts as an adapter layer,
to connect the functions required by RPC to the form in which they
are available from the operating system.
They also perform functions which depend on whether the
communication medium is basically connection oriented or connectionless.

Where the underlying service is a datagram service, the TS layer must
provide such functions as:
<ul>
<li>Reliability
<li>Segmentation
</ul>
For connection oriented communication, those functions are normally
already covered, but the TS layer must now handle:
<ul>
<li>Listening for connections (multiclient server)
<li>Acceptance of connections
<li>Actions when peer disconnects.
<li>Message boundary marking (if stream oriented)
</ul>

The connectionless case is in principle more complicated, but the
more complicated operating system interface for the connection oriented
cases means that neither form of the TS layer is trivial.

<h2>Connectionless communication protocol
<i1>Protocol, datagram
<i1 ix=1>Protocol, datagram
The RPC system was originally designed to run using standard protocols,
all of which were connection oriented.
The built&hyphen.in protocol which was supposed to bridge the gap until
these services were available was designed to be as simpple as possible.

It does not handle segmentation at all. This means that in any
one call over ethernet, the total data in the call or the reply
cannot exceed one packet size (1500 bytes).
For most applications, this was not considered a problem.

The TS layer does use a simple protocol to give reliability in the case
of lost packets.

The assumption is made that the medium is only rarely unreliable
<fn>
This is in practice the case with ethernet, or local RS232 connections.
In our experience to date, packets over ethernet had only been lost
due to receiver overrun (a software problem), but not due to network
saturation or packet corruption.
For several months, RPC systems ran at CERN assuming complete reliability
of the ethernet, without practical problems, before the protocol
was introduced.
</fn>
so no effort was made to make the recovery fast.

Different forms of reliability have been considered for RPC systems,
different "call semantics". These are:
<i1>Call semantics
<ul>
<li>"At most once":
This is basically unreliable remote procedure call, in which
one call is sent out, and if it or the reply is lost, no action
is taken.
<li>"At least once":
In this case, one reply is required, and retransmissions are made,
but no check is made that
a call is not executed twice due to a retransmission.
This is fine for idempotent
<fn>
<i1>Idempotent, definition
<i1 ix=1>Idempotent, definition
An idempotent opertation is one which may be executed any number (>0) of times,
with the same result. Functions without side effects are idempotent,
as are procedures which set a global state to a specified absolute value.
</fn>operations.
<li>"Exactly once":
This is the correct semantics for a remote call which is to emulate a
local call, idempotent or not.
</ul>
The TS layer keeps a notional "connection" between the server and
any particular client, for the purposes of removing duplicate calls.
The protocol used provides "exactly once" semantics except for the first
call any connection, for which "at least once" semantics hold.
The first call is normally an internal idempotent call (to look up the
package number), so this is not a problem.

A summary of the protocol is as follows:
<ul>
<li>Calls sent over a connection are numbered, starting at zero,
then cycling between 1 and some maximum number.
A call numbered zero is always acceptable - this restarts
the sequence (starts a new "connection").
If ever such a packet is received, it is assumed that it is
a new instantiation of a client which has sent it.
<li>If the client does not receive a reply, it retransmits the call
(with the same sequence number).
The retransmission is done after a certain time, which doubles each
retransmission ("exponential backoff").
<li>If the client receives a reply (return or reject) with the wrong
sequence number, it ignores it.
<li>
If it receives a call with (nonzero) number matching that of the last
rely sent, the server retransmits the last reply.
Any other call is executed and replied to.
<li>A multi&hyphen.client server keeps connections (i.e. a record
of the last reply sent) indefinitely.
If there were a very large number of possible clients, this
would obviously eventully cause the server to run out of
memory.
In any case, if TS_Close is called, all messages are returned to the pool.
</ul>

<i1>Retransmission of packets
This is not a perfect protocol, but is was felt that it was not worth the
effort investing in a more complex one unless a standard existed.
Aspects of its simplicity which allow easier implementation include:
<ul>
<li>The client is the only side which retransmits on timeout.
The server is basically passive.
<li>The server is not required to receive messages while executing the remote
procedure. This allows implementaions on strictly single&hyphen.task systems.
</ul>
The simplicity of the protocol gives rise to the following restrictions:
<ul>
<li>The first call must be idempotent.
<li>Large numbers of clients could make a server run out of memory
<li>There is no way for a client to distinguish between a call which
is taking a long time to execute, and a crashed server.
</ul>
(It is important that the protocol should run on a single task system.
In practice, an inefficiency can arrise on such systems, in that
ethernet drivers, when there are insufficient buffers, tend to throw
away the most recent, rather than the latest, packets.
This means that while a procedure is running (and therefore no
servicing of incoming packets is possible, that the buffers will
fill up with retransmitted calls.
In some cases, if the client is faster than the server, then
when the server responds, it loses the next call before it can
clear out the backlog of retransmitted packets from the last call.
This slows the system down by one retransmission on each call following
a call which took a long time.)
<H1>Software Interfaces and Flow Charts
<i1>Interfaces, software, details
<i1>Flow charts
This chapter outlines the two interfaces: that between the stub and the
run&hyphen.time support package, and that between the RTS and the transport
service.

Note that strings are all passed as 40 character arrays, by address. This means
<i1>Strings passed to RTS
that the caller must provide a 40 character string, blank filled if necessary. 
When using FORTRAN on the VAX, the string parameter must be passed using the
built&hyphen.in dummy function %REF(thestring) in order to pass the string by address
rather than by descriptor. The string length (currently 40), for pascal
programmers, is defined in the standard include files as RPC_name_length. 

The definition of the calling conventions for the routines below must
be taken as being the include files which are part of the implementation.
These define the calls in terms of the relevant Pascal compiler
(VAR or not VAR, etc).
Those who program in other languages should ensure that the
calling method they use corresponds.

The "message" type is defined in the standard RPCTYPES include file
issued with the RPC library. This is a Pascal include file,
or C include file, depending on the system.
Anyone writing code which needs to know the format of this message
should either write in Pascal or C, or automate the generation of
their include file from the pascal one.

<hp1>In the flow charts, boxes drawn with heavy lines indicate procedures
expanded elsewhere.
</hp1>
<H2>Run&hyphen.Time Support Calling Sequences

<i1 see="user manual">Calling sequence, for RTS
The calling sequences for these routines are given in the User Manual,
which should be used for reference.
<H3>RPC_Open
<i1>RPC_Open

Called by client stub, at initialisation.

The service_name describes the remote service required. It may be a formatted
physical address (program number, network address, etc) or it may
be a logical name which is converted by the run&hyphen time system into
an appropriate address.

The handle returned is used in all future reference to the
service by the client.

May be called by the user application if he manages his own handles.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip1
<PICTURE NAME=FLOW1$$S>
...skip1
<FIGCAP><HP1>Flow chart of RPC_Open.
</HP1>
</FIG>
RPC_Open, if successful, creates a new record on the client_list.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip29
<PICTURE NAME=FLOW29$S>
...skip29
<FIGCAP><HP1>Format of client_list entries
</HP1>
</FIG>
The work of setting up a client handle is done by the procedure "Setup"
which is also used by RPC_Switch.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip2A
<PICTURE NAME=FLOW2A$S>
...skip2A
<FIGCAP><HP1>Flow chart of Setup (part 1)
<i1>Setup, flow chart
</HP1>
</FIG>
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip2A
<PICTURE NAME=FLOW2B$S>
...skip2B
<FIGCAP><HP1>Flow chart of Setup (part 2)
<i1>Setup, flow chart
</HP1>
</FIG>
Find_Remote is a routine which interrogates the remote node to find
out the package number of a package for which it only has the name.
The call to Find_Remote is in fact a remote call to Find_Local,
and is implemented by a standard stub.
The stub is written into the RPCRTS module explicitly, as if it were
not it would have mutual references with RPCRTS, which would complicate
the implementation under strongly hierarchical systems such as TurboPascal.

<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip21
<PICTURE NAME=FLOW21$S>
...skip21
<FIGCAP><HP1>
Find_Local simply looks up the package name and returns the package number
</HP1>
</FIG>
<H3>RPC_Close
<i1>RPC_Close

Called by client set&hyphen.up when the service is no longer required.
The handle value becomes invalid for future use.

May be called by the user application if he manages his own handles.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip3
<PICTURE NAME=FLOW3$$S>
...skip3
<FIGCAP><HP1>Flow chart of RPC_Close
</HP1>
</FIG>
<H3>RPC_Call
<i1>RPC_Call
Called by client stub.

On entry, the buffer consists of a header (in which the procedure number
has been written) followed the procedure arguments.
The count tx_bytes includes only the
number of bytes in the procedure argument array:
it does not include the header.
The timeout parameter specifies
(nominally in 10us units)
a maximum time allowed for the call.
A negative value disabled the timeout (allows an infinite time).

The buffer_size parameter indicates the total size in the buffer
available for returned arguments (again, exclusive of the header).

There are two versions of this routine. RPC_Call itself handles errors,
whereas RPC_Call_Status returned error status to te caller.

A possibility here is for the RTS to deallocate the buffer, and
pass back a pointer to a different one used for recieving the data.
This saves time in certain cases, especially if the
communications service can accept a user buffer and deallocate it
itself after the data has left.
Therefore, the caller of RPC_Call must not keep any other record of the
address of his buffer.
..us off

<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip4A
<PICTURE NAME=FLOW4A$S>
...skip4A
<FIGCAP><HP1>Flow chart of RPC_Call and RPC_Call_Status.
In the case of a local call,
the routine Handle_Call is used. RPC_Call_Status returns the status
value at the end of this part, wheras RPC_Call continues to treat the
error.
</HP1>
</FIG>
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip4B
<PICTURE NAME=FLOW4B$S>
...skip4B
<FIGCAP><HP1>Flow chart of RPC_Call (continued).
</HP1>
</FIG>
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip5
<PICTURE NAME=FLOW5$$S>
...skip5
<FIGCAP><HP1>Flow chart of Handle Call. This routine is used for
local calls as well as by a server of remote calls.
</HP1>
</FIG>
<H3>RPC_Attach_Stub
<i1>RPC_Attach_Stub

Called by server stub on initialisation.
The entry_point parameter passes the actual stub procedure to be attached.
The service_name parameter gives the logical name of the server.
This may be used by the run&hyphen.time support to update name tables.
The service name will be the same for different instances of the same server.
The program number returned may be used to disconnect the stub later.

Internally, this calls RPC_Specify_Stub.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip6
<PICTURE NAME=FLOW6$$S>
...skip6
<FIGCAP><HP1>Flow chart of RPC_Attach_Stub
</HP1>
</FIG>
This procedure sets up a new server stub record on the list "program_list".
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip28
<PICTURE NAME=FLOW28$S>
...skip28
<FIGCAP><HP1>Format of program_list entries
</HP1>
</FIG>
<H3>RPC_Specify_Stub
<i1>RPC_Specify_Stub

This is identical to RPC_Attach_Stub, except the program number is
generated by the user.
This allows him to allocate "well known numbers" to represent
particular packages.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip7
<PICTURE NAME=FLOW7$$S>
...skip7
<FIGCAP><HP1>Flow chart of RPC_Specify_Stub
</HP1>
</FIG>
<H3>RPC_Detach_Stub
<i1>RPC_Detach_Stub

Called by server set&hyphen.up when the server is no longer required.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip8
<PICTURE NAME=FLOW8$$S>
...skip8
<FIGCAP><HP1>Flow chart of RPC_Detach_Stub
</HP1>
</FIG>

<H3>A Server Stub
<i1>Server stub, interface

        IN      call_buffer:    pointer to message

..bx off
Called by the RTS.
The length of the stub buffer should be obvious to the stub code,
which is aware of its format.
In case of error, the stub writes a bad status into the
m_status field of the message.

<H3>RPC_Early_Return
<i1>RPC_Early_Return
Called by the server stub, only in the case of a "concurrent" procedure.
The message consists of a header (which is left intact by the stub)
and a results array, which is filled.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip9
<PICTURE NAME=FLOW9$$S>
...skip9
<FIGCAP><HP1>Flow chart of RPC_Early_Return
</HP1>
</FIG>
<H3>RPC_Loop_Server
<i1>RPC_Loop_Server
Called by: User server program.

The set of clients to be serviced is identified by a string, which
may in some cases include wildcards.  This procedure is the simplest form
of server, as it loops forever, servicing incoming requests.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip10
<PICTURE NAME=FLOW10$S>
...skip10
<FIGCAP><HP1>Flow chart of RPC_Loop_Server
</HP1>
</FIG>
<H3>RPC_Start_Server
<i1>RPC_Start_Server
Called by: User server program.

The set of clients to be serviced is identified by a string, which
may in some cases include wildcards.  This procedure queues a read request,
and has the next message and successive messages serviced at AST level.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip22
<PICTURE NAME=FLOW22$S>
...skip22
<FIGCAP><HP1>Flow chart of RPC_Start_Server
</HP1>
</FIG>
RPC_Start_Server specifies, as the AST procedure to run, RPC_Server_AST.
This routine first handles the call, and then queues the next read on
the channel. Originally, it used to queue the read in advance, which would
speed up the system in some cases, but this causes problems if sophisticated
stubs or user routines use the same communication channel while the call
is being handled: messages would be received in the wrong order.
(See the PIPELINE precompilation option).
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip30
<PICTURE NAME=FLOW30$S>
...skip30
<FIGCAP><HP1>Flow chart of RPC_Server_AST
</HP1>
</FIG>
<H3>RPC_Create_Server
<i1>RPC_Create_Server

This routine sets up the communication channel for a server, but does not
let it run.  The socket returned may be used with the routines below
to service requests as required.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip20
<PICTURE NAME=FLOW20$S>
...skip20
<FIGCAP><HP1>Flow chart of RPC_Create_Server
</HP1>
</FIG>
<H3>RPC_Accept
<i1>RPC_Accept
This routine waits for a request on the given socket, and services it
according to the currently attached stub modules.
The timeout is in units of 10ms.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip11
<PICTURE NAME=FLOW11$S>
...skip11
<FIGCAP><HP1>Flow chart of RPC_Accept
</HP1>
</FIG>
<H3>RPC_Queue_Server
<i1>RPC_Queue_Server
This routine queues an asynchonous user routine (AST) to be called when the
next request arrives on the given socket (see RPC_create_server). It allocates
space for the incoming message, and passes the address of this to the AST
routine.
The user AST
routine has entry point astadr and is passed by address one parameter (info).
This parameter is suitable to be passed to RPC_Service (see below).
Its format is private to the RPC system.

This parameter is all that is required by the service routine below: the user
program may, however, want to perform some pre&hyphen. or post&hyphen. processing, such
as forking off other tasks to handle the requests, or application functions.

(For the format of the message buffer, see the rpc$types include file,
as this contains several fields used internally by the RPC RTS, which may be
changed between versions.  It is not recommended that user code uses this
buffer directly, but calls RPC_service to have it serviced.)

This routine, with RPC_Service, was used to make
a multitask server under the MODEL TCS scheduler.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip12
<PICTURE NAME=FLOW12$S>
...skip12
<FIGCAP><HP1>Flow chart of RPC_Queue_Server
</HP1>
</FIG>
<H3>RPC_Service
<i1>RPC_Service
This is a procedure taking one record parameter as described in the preceding
paragraph. It will handle one incoming call message, deallocate the message
buffer passed to it, and return. 
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip13
<PICTURE NAME=FLOW13$S>
...skip13
<FIGCAP><HP1>Flow chart of RPC_Service
</HP1>
</FIG>
<H3>RPC_Configure
<i1>RPC_Configure
This call allows one client to use more than one identical server in turn.
The effect is for the connections to all open servers to be closed and reopened
with the translation of their logical names being perfomed according to
new values which may have been set up.

A client process which wishes to use several identical servers may therefore
repeatedly redefine the logicalnames which give the addresses of those servers,
and call RPC_configure to cause future calls to be directed to the new
servers. 
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip14
<PICTURE NAME=FLOW14$S>
...skip14
<FIGCAP><HP1>Flow chart of RPC_Configure
</HP1>
</FIG>
<H3>RPC_Switch
<i1>RPC_Switch

This performs the reconfiguration function on a single package,
referenced by a handle. The package must be already open
(for the handle to be valid).
The value of the handle is unchanged.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip15
<PICTURE NAME=FLOW15$S>
...skip15
<FIGCAP><HP1>Flow chart of RPC_Switch
</HP1>
</FIG>
<H3>RPC_Establish
<i1>RPC_Establish
This procedure may be used in a client program to handle errors occuring
in remote procedure calls.
By default, if an unrecoverable error occurs in the communication with the
server, the client program is stopped.
($EXIT occues under VAX/VMS, a pascal halt on other systems.)

If the user wishes to handle the error himself, he passes the address of
his error handler to rpc_establish.
If this has been done, his handler will be called
<hp1>instead</hp1>
of the default handler.

In order to revert to the default handler, RPC_establish is called with
a value of zero.

The user error handler is a routine taking one parameter,
which is the message in question.
If the user is programming in pascal, he will find the offending error
status in the m_status field of the message.

RPC_Establish simply places the given value into a global variable.
<h3>RPC_Init
This is called automatically where possible (e.g. VAX/VMS, TurboPascal,
MoniCa).
In other cases, it must be called explicitly by the user program.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip17
<PICTURE NAME=FLOW17$S>
...skip17
<FIGCAP><HP1>Flow chart for RPC_Init
</HP1>
</FIG>
<h2>Internal Routines
These routines are not normally called from outside the run time support.
<h3>RPC_New
<i1>RPC_New
RPC_New is the routine which produces a new message buffer.
It will allocate a message from the pool if the pool is not empty.
Depending on whether the heap is in use, it may use the heap if the pool
is empty.
RPC_New may be called from the transport service as well as from stubs
and internally.
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip18
<PICTURE NAME=FLOW18$S>
...skip18
<FIGCAP><HP1>Flow chart for RPC_New
</HP1>
</FIG>
<h3>RPC_Dispose
<i1>RPC_Dispose, flow chart
RPC_Dispose performs the reverse operation to RPC_New, except that it
never returns a message to the heap, it keeps it for efficiency.
..go zz
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip19
<PICTURE NAME=FLOW19$S>
...skip19
<FIGCAP><HP1>Flow chart for RPC_Dispose
</HP1>
</FIG>
...zz
<h2>TS layer TS_XXXX procedures
<i1>Transport Service, calling sequence
The calls used internally by the run&hyphen.time
system are currently of the form given below. The status code is returned
as a function value in each case.
Outline flow charts are given below 
for some parts of a typical example of a TS adapter layer
(TSPM68k) which supports RS232 and Ethernet communication.
In some cases, the connection oriented communication handling
is also shown.
The reader is refered to the source for the system in question for
the exact parameter passing conventions, data types, etc.

<H3>TS_Open(socket, name)
<i1>TS_Open
<xmp>
                OUT     status:         integer (function value)

                OUT     socket:         integer
                IN      name:           string
</xmp>
All the information about the medium and about the  protocol, service type
and peer address where applicable are coded into
the name string.  A wildcard syntax in the string signifies connection to a
class of peers. In this case the address for transmission is always to be taken
from the source of the last received message. A socket number for reference is
returned. 
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip23A
<PICTURE NAME=FLOW23AS>
...skip23A
<FIGCAP><HP1>Flow chart for TS_Open (part 1)
</HP1>
</FIG>
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip23B
<PICTURE NAME=FLOW23BS>
...skip23B
<FIGCAP><HP1>Flow chart for TS_Open (part 2)
</HP1>
</FIG>
<H3>TS_Close(socket)
<i1>TS_Close
<xmp>
                OUT     status:         integer (function value)

                IN      socket:         integer 
</xmp>
The socket number becomes unusable in future calls.

Any messages attached associated with the socket are returned to the
message pool,
<hp1>including any passed by the user as a parameter to an oustanding
TS_When_Receive request.</hp1>
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip24
<PICTURE NAME=FLOW24$S>
...skip24
<FIGCAP><HP1>Flow chart for TS_Close
</HP1>
</FIG>
<H3>TS_Send(socket, data, length) 
<i1>TS_Send
<xmp>
..tb 9 17 25 33 41 49 57 65 73
                OUT     status:         integer (function value)

                IN      socket:         integer
                IN      data:           pointer to message
                IN      length:         integer
</xmp>
The socket passed must be derived from TS_Open.  The data array
is in fact a data structure of which the first part is internally
used by the RPC system, the second part is free for use by the
transport service, and the third part is data belonging to the
RPC system, which is actually to be transferred.

<dl>
<dt>On entry
<dd>
Within the first part, two of the four fields are set up
before TS_send is called:
<xmp>
..tb 9 17 25 33 41 49 57 65 73
        M_Next          is undefined
        M_Index         must be the length of the message to be sent
        M_Socket        must point to the socket to be used.
        M_Status        is undefined
</xmp>
(Note that this information duplicates, for historical reasons, data in
the parameters 'socket' and 'length'. In practice, these fields
have turned out more useful than the parameters, so if the system
were designed again, TS_SEND would only take one parameter.)

<dt>On exit
<dd>The pointer to the data may have been changed, so as to point to
another message.
This allows copying to be reduced, for speed.
In the message pointed to be the (new) pointer,
<xmp>
..tb 9 17 25 33 41 49 57 65 73
        M_Next          may be corrupted
        M_Index         may be corrupted (currently)
        M_Socket        points to the socket to be used.
        M_Status        may be corrupted
</xmp>
</dl>
<H3>TS_Receive(socket, data, max_length, xfer_count)
<i1>TS_Receive
<xmp>
..tb 9 17 25 33 41 49 57 65 73
                OUT     status:         integer (function value)

                IN      socket:         integer
                IN OUT  data:           RPC_message_pointer
                IN      max_length:     integer
                IN      timeout         integer
</xmp>
The socket passed must be derived from TS_Open.  The data array is in fact a
data structure of which the first part is internally used by the RPC system,
the second part is free for use by the transport service, and the third part is
data belonging to the RPC system, which is actually to be transferred. 

A pointer to this data structure is passed by address. The ts_receive routine
is free to switch it to a different buffer, to save time. The caller must not,
therefore, keep any other record of the location of the original buffer. 
If the buffer pointer is switched, ts_receive will dispose of (or do something
intelligent with) the old buffer.

The timeout parameter specifies
(nominally in 10us units)
a maximum time allowed before reception.
A negative value disables the timeout (allows an infinite time).
(The unit of 10ms is not a very friendly one, but was chosen
in order to give a sufficiently long maximum timeout on a 16 bit
system).
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip25a
<PICTURE NAME=FLOW25AS>
...skip25a
<FIGCAP><HP1>Flow chart for TS_Receive (part 1)
</HP1>
</FIG>
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip25b
<PICTURE NAME=FLOW25BS>
...skip25b
<FIGCAP><HP1>Flow chart for TS_Receive (part 2)
The decision "Duplicate?" in th centre of this chart indicated a call
to the function "Duplicate" or "Duplicate_V24".
</HP1>
</FIG>
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip27
<PICTURE NAME=FLOW27$S>
...skip27
<FIGCAP><HP1>Flow chart for Duplicate and Duplicate_V24.
Areas of difference between these two modules are
indiated by background shading.
</HP1>
</FIG>
<box>
The routines below are not necessary in an implementation which is not
to support asynchronous servers.
</box>
<H3>TS_When_Receive(socket, pdata, max_length, proc, param)
<i1>TS_When_Receive
<xmp>
..tb 9 17 25 33 41 49 57 65 73
                OUT     status:         integer (function value)
                IN      socket:         integer
                IN      pdata:          pointer to message
                IN      length:         integer
                IN      proc:           service_procedure (see below)
                IN      user_param:     integer
</xmp>
The socket passed must be derived from TS_Open.  The data array
is in fact a data structure of which the first part is internally
used by the RPC system, the second part is free for use by the
transport service, and the third part is data belonging to the
RPC system, which is actually to be transferred.

This is a receive subroutine needed for asynchronous
processing of events.  It asks for the procedure "proc"
to be called when the next message arrives.
The address of a suitable buffer is passed, and its
size, and the entry point of the procedure to be called when the read completes.
<dl>
<dt>On entry
<dd>
Within the first message, one of the first four fields is set up:
<xmp>
..tb 9 17 25 33 41 49 57 65 73
        M_Next          is undefined
        M_Index         is undefined
        M_Socket        must point to the socket to be used.
        M_Status        is undefined
</xmp>
<dt>On exit
<dd>The pointer to the data may have been changed, so as to point to
another message.
This allows copying to be reduced, for speed.
In the message pointed to be the (new) pointer,
<xmp>
..tb 9 17 25 33 41 49 57 65 73
        M_Next          may be corrupted
        M_Index         may be corrupted
        M_Socket        points to the socket to be used.
        M_Status        may be corrupted
</xmp>
</dl>
The asynchronous service procedure referred to ('proc') takes parameter
<xmp>
..tb 9 17 25 33 41 49 57 65 73
                IN      pdata:          pointer to message.
</xmp>
Within that message (which is not necessarily the same message as was
originally passed to TS_When_Receieve), when the 'proc' is called,
<xmp>
..tb 9 17 25 33 41 49 57 65 73
        M_Next          is undefined
        M_Status        is the completion status of the read
        M_Socket        is the socket on which the message arrived
        M_Index         is undefined
</xmp>
<FIG PLACE=FLOAT FRAME=NONE>
..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip26
<PICTURE NAME=FLOW26$S>
...skip26
<FIGCAP><HP1>Flow chart for TS_When_Receive
</HP1>
</FIG>

<H3>TS_My_Address(socket, name)
<i1>TS_My_Address
This returns the address at which the user may be contacted.
<xmp>
..tb 9 17 25 33 41 49 57 65 73
                OUT     status:         integer
                IN      socket:         integer
                OUT     name:           string
</xmp> 
This is not a very well defined function, in that what can be returned,
and it's exact significance, depends on the transport medium being used.
Over a serial line, it is impossible to know by what name the line
is known at the far end.
Over ethernet, though, it is very useful to be able to find one's own
ethernet address.
In general, the address is returned in the format of an RPC address,
as used for TS_Open.
<H3>TS_Peer_Address(socket, name)
<i1>Address of peer
<i1>Peer address
This returns the address of the sender of the last received message.
<xmp>
..tb 9 17 25 33 41 49 57 65 73
                OUT     status:         integer
                IN      socket:         integer
                OUT     name:           string
</xmp>
The same problems exist as for TS_My_Address, in that the information
which is available and its significance tend to vary between systems.
Again, the rule is that an address which would work with TS_Open
should be returned.
Over DECNET, for example, one can find the task name but not the TSAP name:
in this case an address is returned which would work if the peer task had
created a TSAP with the same name as his process.
<h2>LANCE ethernet chip routines
These routines are written in C.
(Files tlance.c, lance.h)
They were derived from routines
written by L.R.T. Ltd, modified by Ben Segal (CERN/DD/SW),
and further modified for inclusion in the RPC project.
They do not use the data chaining features of the lance, as RPC buffers
are always sufficiently large for a packet.
They are aware, on the receive side, of the TS layer's socket
format, as they search the socket list for a socket which is
waiting for the received packet.

These routines have survived a number of releases of the LANCE chip which
had more or less obscure bugs. Therefore, they are fairly defensive,
and will reinitialise the chip if things go wrong.

There is a reasonable amount of trace code built into the module,
controleed by an integer variable _debug taking values from 0 to 5
(see the user manual).
Trace output from this module is prefixed by "* LANCE: ".

The entry points are as follows. Note that as a peculiarity of the C
compiler, they must be prefixed by an underscore character if referenced
from outside the C world.
<h3>bbinit
This is a one-time general initialisation.
It sets upo data structures in memory, initialises the chip itself,
and sets up (via a small piece of assembler) the interrupt vectors
to point (via a small piece of assembler) to the
..bf MONO
lance_i
..pf
interrupt service procedure.
On exit from _bbinit, the lance chip is initialised by inactive.
<h3>bbopen
This routine activates the lance chip.
For safety, it resets the buffer system.
<h3>bbclose
This routine deactivates the LANCE chip.
<h3>lance_send
This routine takes as one parameter the address of a packet to send.
It returns a status when it has succeded, or faileded after several ettempts,
to send the packet.

The following entry points are used internally to the tlance.c module.

<h3>lance_i
This is the general interrupt service routine.
It displays trace information if errors occur, if the level of _debug
is high enough.
It calls lance_read if a packets has been received.
<h3>lance_read
This checks that a packet really has arrived, and that it is good.
If so, it scans the queue of sockets waiting for ethernet packets
looking for an address and protocol type match.
A mask on this check allows wildcard reception by a socket.
<h3>getstation
This routine is used internally to extract the ethernet address of the
card in use from the hardware.
<h3>initilise_lance
This sets up the data structures and initialises the chip.
<H1>Message Formats

<i1>Message Format
The messages are formatted according to the Courier
<BIBREF REFID=COURIER>
format.
The "extended subset" of that format which is used is described below.
The message is described as a stream of 8 bit bytes.
When word&hyphen.oriented communication is used, one must ensure
that the byte order is preserved.

An alternative representation considered was
the X.409 standard
<i1>X.409
<BIBREF REFID="X409">.
It requires more bytes, and so is not appropriate in bandwidth&hyphen.limited
systems such as the Valet+, but is becoming accepted as a standard
for data representation.
It may be supported as an alternative.

A further alternative may be the Apollo NCS format, purely for
compatibility with NCS.
<i1>Apollo NCS
<H2>Data type representation

<i1>Data representation
<i1>Representation, data
A list of the data types handled by the RPC system are given in
appendix A of the User Manual.
Below is listed the way each data type is packed into the message.
Bytes (represented by boxes) are sent in the order listed left to right
across the page.
Note that each type is padded with zeroes so that every element
wil start on a 16 bit boundary, according to the Courier spec.
This may be suppressed, for efficiency, using the NOALIGN option
in the INCLUDE file after running the RPCC and before running the
Pascal compiler on the stubs.

..tb 9 17 25 33 41 49 57 65 73
..bd RPC_BYTE
<i1>RPC_Byte

..cc on
..tb set $
..bx 8 15 / 16 23
$0$Value
..bx off

..bx 8 15
$Value$  (If NOALIGN option used)
..bx off
..cc off

..bd RPC_CHAR
<i1>RPC_Char

This is sent as an ASCII (or, rather, CCITT International Alphabet 5) character.
Translation is performed by the stubs if necessary.

..cc on
..tb 9 17 25 33 41 49 57 65 73
..bx 8 15 / 16 23
$0$Value
..bx off

..bx 8 15
$Value$$(If NOALIGN option used)
..bx off
..cc off

..bd RPC_SHORT
<i1>RPC_Short

This may be used to represent a signed (2s complement) or unsigned
quantity. (The two's complement notation is so standard that
it is not necessary to distinguish between them at this level.
If any machine used a different format for negative numbers, we would
have to).

..cc on
..bx 8 15 / 16 23
$MSB b1$LSB b0
..bx off
..cc off

Note that the MOST SIGNIFICANT BYTE IS SENT FIRST.  If the number is
given by

        X = b0 + (256 * b1)

then the order of bytes in the buffer is b1, b0 as shown.

..bd RPC_LONG
<i1>RPC_Long

This may be used to represent a signed (2s complement) or unsigned
quantity. (The two's complement notation is so standard that
it is not necessary to distinguish between them at this level.
If any machine used a different format for negative numbers).

..cc on
..bx 8 15 / 16 23 / 24 31 / 32 39
$MSB b3$b2$b1$LSB b0
..bx off
..cc off

Note that the MOST SIGNIFICANT BYTE IS SENT FIRST.  If the number is
given by

        X = b0 + (256 * b1) + (65536 + b2)  +  (256*65536* b3)

then the order of bytes in the buffer is b3, b2, b1, b0 as shown.

..bd RPC_REAL32
<i1>RPC_Real32

This is sent in IEEE format.
(There is no defined format for real numbers in either the Courier
or the X.409 recommendation).
This, and conversion to and from other 32 bit formats is discussed in
<BIBREF REFID="REALS">
Conversion is performed, on the VAX, when the number is (un)packed by the
stub.

..cc on
..bx 8 10 15 / 16 18 23 / 24 31 / 32 39
$S  exp$e  MSB$mantissa$LSB
..bx off
..cc off

The first byte sent contains the sign bit (S) in the most significant bit,
and then the 7 most significant bits of the exponent.
The next byte contains the least significant bit of the exponent (E),
and the 7 most significant bits of the mantissa.
The next two bytes contain the rest of the mantissa.

The sign bit, if set, means that the number is negative, though
the mantissa remains uncomplemented.
The exponent field is zero to indicate a zero real number.
The mantissa is unsigned and normalised, with the leading "1" omitted.
The exponent bias is 150. The absolute value of a number is

        X = 2**(exponent-150) * (mantissa + 2**23)

..BD RPC_STRING
<i1>RPC_String

This is represented by a two byte length field, followed by
a series of one&hyphen.byte characters, followed by a zero byte
if this is necessary to leave the total number of bytes even.

The lengh field is in RPC_SHORT format.
The characters are in the same coding as RPC_CHAR,
with conversion being done by the stubs as necessary.

..cc on
..bx 8 15 / 16 23 / 24 31 / 32 39 / 48 55 / 56 63
$Len b1$Len b0$Char 1$Char 2$...$Char n$Zero
..bx off
..cc off

In this example the characters are char 1.. char n, where n
is odd, so a padding zero byte is required.

..bd Arrays and Records
<i1>Arrays, representation
<i1>Records, representation

Arrays are of fixed, predefined, size, and are packed simply
as a series of elements of thier component type.
A multidimensional array is treated as an array of arrays,
so the last index cycles most rapidly.

A record is represented simply by the set of its constituent fields,
in the order in which they were declared.

..bd Sequences
<i1>Sequences, representation

Sequences, in RPCL, are arrays of length unknown at compile time.
They are transferred preceded by a count field, in RPC_INTEGER
format.
..tb set
<H2>X.409 Alternatives
<i1>X.409 mapping
In an alternative representation, data is represented in X.409 format
<BIBREF REFID="X409">
In this case, the message is a series of bytes ("octets"),
there being no word alignment.
Every data item it represented by an identifying byte, followed by
a length field of one or more bytes, followed by the data.
The RPCL types map onto X.409 types as follows:

        RPC_BYTE, RPC_SHORT, RPC_LONG:
                Integer (Id 02 hex)

        RPC_CHAR, RPC_STRING:
                IA5 string (Id 16 hex)

        ARRAY, SEQUENCE:
                Sequence (ID 30 hex)

Using tagging in this way requires more run&hyphen.time code and
execution time, but allows makes a message more readable by
a third party unaware of its formatting.

<H2>Message content

Below we list the contents of a message of each type.
Every message begins with a "message type" field, of type RPC_SHORT,
which can take the values

        0       Call message
        1       Reject message
        2       Return message

There follows a transaction identifier(a 16 bit number) which may be
used for sequence checking, but at present is always set to zero.

The formats of each type of message are as follows:

<H3>Call message
<xmp>
        message_type            RPC_short       0
        transaction_id          RPC_short
        package_number          RPC_long
        version_number          RPC_short
        procedure_number        RPC_short       (within the package)
</xmp>
This header is followed by all the IN and INOUT parameters,
in the order specified in the definition file.

The transaction ID is only used when the simple protocol is
run over a connectionless communications service.

The package number and version number are described in the
user manual. The procedure number is the index of the procedure
within the package, generated by the RPC compiler.
<H3>Reject message
<xmp>
        message_type            RPC_short       1
        transaction_id          RPC_short
        reason                  RPC_short       Courier reason number
        status                  RPC_long        CERN (VMS like) error code.
</xmp>

<H3>Return message
<xmp>
        message_type            RPC_short       2
        transaction_id          RPC_short
</xmp>
This header is followed by the function result (if any),
then by all the OUT and INOUT parameters,
in the order specified in the definition file.

<h2>Coding scheme for RS232 point&hyphen.point messages 

<i1 see="RS232">V24
<i1>RS232 transmission encoding
There are two levels of encoding: the frame formatting
(header and checksum), and the byte encoding.

In order to send binary byte stream data over
7&hyphen.bit ASCII terminal channels, an
ad hoc coding scheme has been implemented, which is described below. The scheme
is neither theoretically optimal nor beautiful nor standard, but is simple and
relatively efficient. It is designed to get though the filters and conditions
imposed on such channels by operating systems and drivers. 
This coding scheme is optional, and used only if 7 bit communication
is requested.

The byte encoding scheme allows 8&hyphen.bit bytes to be logically sent.
In some cases, they can be sent directly, where both transmitting
and receiving operating systems allow it.
Where this is not so, a seven bit encoding scheme
satisfies the following requirements:
<ul>
<li>Only seven bit characters are used
<li>Control characters (00 to 1F hex) are not used
<li>The DELETE character is not used
<li>Small vales are transmitted as single bytes
</ul>

<H3>8&hyphen.bit encoding scheme for bytes

In this case, the only formatting necessary is the escaping
of the start of message character, and of the escape character.
<H3>7&hyphen.bit Encoding scheme for bytes.
This scheme was updated by RB in summer 1988.

When transmitting a byte,
<ol>
<li>If the byte is less than 5C hex, add 20 hex to it and transmit it.
<li>If the byte is between 5D and B9 hex inclusive, precede it with
the character 7D, then subtract 3D from it and send the result.
<li>If the byte is between BA and FF hex inclusive, precede it with
the character 7E, then subtract 9A from the byte and send the result.
</ol>
<xmp>
        x in  0..5C             -->     x-20
        x in 5D..B9             -->     7D, x-3D
        x in BA..FF             -->     7E, x-9A
</xmp>
When receiving it, the reverse operation is
<ol>
<li>If the character is less than 7D, subtract 20 from it.
<li>If the character is 7D, take the next one and add 3D.
<li>If the character is 7E, take the next one and add 9A
</ol>
<H3>Framing format

There are two frame formats.
One is a general one, which can reprsent any RPC message.
The other is a specially efficient one optimised
for a strict subset of possible messages, which will
suffice for simple systems.

The frame starts with
<xmp>
                  Message Type,
                  Transaction ID,
</xmp>
which, for a Call type message, should be followed by
<xmp>
                  Program Number,
                  Version Number,
                  Procedure Number,
                  all the IN and INOUT parameters,
</xmp>
The following encoding scheme is used:
<xmp>
            7   6   5   4   3                    2   1   0
          _____________________                _____________
          | M : T : y : p : e |                | T : I : D |
          ---------------------                -------------
           0 - Call, long format               Packet Number
           1 - Reject
           2 - Return
           4 - ? (Cast)
           5 - ?
           6 - ?
           7 - ?
           8 - Call Program    0, short format
           9 - Call Program    1, short format
           n - Call Program  n-8, short format
          31 - Call Program   23, short format
</xmp>
This mechanism,
allows us to send both the Message Type
and Transaction ID in just one byte.

In the case where

<xmp>
..tb 9 17 25 33 41 49 57 65 73
               The message is a CALL    and
               Program Number   <  24   and
               Version Number   =   0   and
               Procedure Number < 256,                         (1)
</xmp>
the short form is used.
In this case there are just two bytes to send:
<xmp>
..tb 9 17 25 33 41 49 57 65 73
                The byte above   (1 byte)
                Procedure number (2 bytes)      Total: 3 bytes
</xmp>
If the condition (1) is false we must use the long header format
that requires us to send:
<xmp>
..tb 9 17 25 33 41 49 57 65 73
                The byte above   (1 byte),
                Program Number   (4 bytes),
                Version Number   (2 bytes),
                Procedure Number (2 bytes).     Total: 9 bytes
</xmp>
The equivalent information is stored in memory, and sent over
ethernet, etc., as 12 bytes.

<i1>Frame format, over RS232
<i1>Short frame format
The frame is preceded by a frame start byte,
and followed by a checksum.
A one byte checksum is used for messages below (???) bytes in length,
and for longer messages a 32 byte CRC is used.

<H3>Frame reception
Any control characters received should be ignored at any time.

Any characters received before the start of frame character
shall be ignored.

<H1>Porting the RPC system into a new environment

<i1>Porting the system
If a new machine or software environment is to be introduced to a distributed
system using RPC, there are three stages of porting of software wich may be
necessary. These are the provision of communications support in the new
environment, the porting of the RPC RTS itself, and, if required, the porting
of the compiler.

<H2>Provision of the Communications support

<i1>Transport Service, porting
<i1 see="Transport Service">Communications
Assuming that communications support is available for a suitable medium
in the new environment, all that must be done is to provide it in the
(relatively simple) form needed by the RTS.   In the simplest case
(with no asynchronous calls) all that is required are the OPEN, CLOSE,
SEND and RECEIVE procedures.

One of the existing TSxxx modules may be taken as a template, and
used
<ol>
<li>as an adapter to the existing communications primitives
<li>as a switcher to select the communication medium required.
</ol>
<H2>Porting the Run&hyphen.Time System

<i1>RPCRTS, porting
This module is written in Pascal, with conditional compilation sections
handled by the INCLUDE facility.  Porting onto a new environment may
require slight additions to the master source file if the Pascal
compiler on the new machine has eccentricities not already catered for.

A version of the RTS now exists which has been translated into C.
This is generated by slightly preprocessing a version of the pascal source,
running the 'ptc' converter progrem, and then
slightly postprocessing the C source it produces.

Therefore, whereas a TS module exists for each target environment, there
is only one master RPCRTS file.

<H2>Porting the Compiler

<i1>Compiler, porting the
The same applies to the compiler: it is written in Pascal, with conditional
compilation sections handled by the INCLUDE facility.  Porting onto a new
environment may require slight additions to the master source file if the
Pascal compiler on the new machine has eccentricities not already catered for.
If cross&hyphen.compilation on an existing machine
(for example VMS or Unix 4.2 Bsd)
is satisfactory, then porting of the compiler itself will not be necessary. 

It may also be the case that modifications have to be introduced into
the code generator if the stub code requires special facilities
for the new environment.

In general, no modifications to the compiler at all will be necessary
if the port is to a new runtime software environment on a CPU
which is already supported.

<h1>Code Management
<i1>Code management
<i1>Source management
<i1>CMS
<i1>MMS
DEC's Code Management System (CMS) is used for all source files,
and the Module Management System (MMS) is used to control the
generation of code and avoid inconsistencies.
Those wishing to change any code in the RPC system should make themselves
familiar with both of these.
They allow, especially when used in a controlled way,
a project to be partially self&hyphen.documenting.

The following notes are provided to help the many people working
together on projects such as the RPC, CHI etc to find their way
around each other's files.
The rules given should make this simpler.
(They are not all adhered to 100% of the time
<h2>Directory management

<i1>Directory management
Any development directory contains:

<dl>
<dt>Source files
<dd>made by hand, cannot be generated automatically.
<dt>Product files
<dd>final products, or files used to
to make products of other directories.
<dt>Byproduct files
<dd>made in the process of making other files but
not needed by any system. (eg maps, listing files).
Not used to make other files
<dT>Intermediate files
<dd>used in the process of making product files,
can be generated automatically.
<dt>Disconnected files
<dd>Not involved at all in the software generation
(e.g. documentation, subdirectories)
</dl>
The use of MMS allows these files to be distinguished by a newcomer to
the project.
Note that the same directory is used for the source and object files:
the subdirectory structure is used to distinguish different areas of the
project, not different types of file.
<ol>
<li>Every directory with any product files MUST contain a makefile
called Makefile (or descrip.mms on VMS).
<i1>Makefile
<li>Any files in a directory which are final products or used in
other directories must be explicitly declared in the Makefile
(or descrip.mms) in that directory.
<li>The default make (MMS) command must bring ALL product files in the
directory up to date. The default target is the first dependency
in the file. For example, it might be
<xmp>
                default : xxx, yyy, zzz
                        touch default
</xmp>
Therefore, one can find out how to make any file required by
any other directory, by looking in the Makefile. One can check
whether the products are up to date with an MMS/SKIP/NOACT
command.
<li>The MAKE CLEAN command (MMS CLEAN) should delete all intermediate files
(except those which take a ridiculous amount of time to build),
and any byproduct files with limited usefulness.

Therefore, it is simple to save space in a directory which someone
else has left full of intermediate files.
<xmp>
        Example:
                clean :
                 -      delete *.lis.*, *.ccu.*, *.lmp.*
</xmp>
<li>The MAKE CLEAN_ALL command should delete all intermediate files,
any product files of which the distibution copies are kept
elsewhere, and all byproduct files.
<li>All source files must be in one (and one only!) CMS library.
If the CMS library is not the defualt one for that project
(say, the first CMS.DIR found going up the directory tree)
then the rule for fetching the source file from the correct CMS
library must be put into the makefile.

Therefore, a modification history for the whole project is
<i1>Modification history
effectively available.
Also, an MMS/CMS command should be able to
rebuild all product files from a directory empty except for the
makefile.
<li>All the software and build mechanisms for a released product
must be under project users (eg RPC, FASTBUS, PILS, VALET).
Any copies under personal users should be experimental only.
</ol>
<h2>C Programming Standards
<h3>File organisation
Experience suggests the following rules
are necessary for portability and comprehensibility:
<ol>
<li>Filenames should have only one dot (for VMS, MSDOS),
and a maximum of 8 characters before the dot (for MSDOS, 9 for VM) and 
3 characters after the dot (MSDOS). (The limit on the Cray is
a maximum of 14 characters including the
dot and the extension.)
<li>
The public include file is the one used by users of the products of
the project. It should be referred to using <angle brackets>.
Other include files used for development within the project is
kept within the project directory and referred to using "double quotes".
Those wishing to use a specific development version of the public
include file may use the -I (unix) or /INCLUDE= (VMS) options of the C
compiler.
<li>For any module which may be called from other modules, 
a .h file should be used to define the entry points and any other
definitions needed by the caller.
This include file have as name the facility name followed by .h.
Any include files used internally, for example for sharing data
or structures between modules, should have a different name to the .c files.
</ol>
<h3>Style
<i1>Style, in C
Many of the rules below are essential for portability and readability,
or compatibility with some utilities,
and are not simple a matter of taste.
However, there are a few questions of personal preference.
When editing anyone's
file, please do not change the layout just to suit yourself,
unless changing it to the common layout below.
The following conventions were adopted in principle for the RPC code,
and should be used for new files.

At the start of a file, the file header should contain:
<ul>
<li>An english title for the file (not its name) underlined with ===.
<li>On the right, ending in column 80, the file name.
<li>After the title, an abstract of the file.
<li>A list authors (names and affiliations) so that later modifications
and personal comments
can be marked simply with initials.
<li>A list <hp2>restrictions</hp2> on the use of the module,
special consitions for its compilations,
limitations as to its portability, etc.
<li>A history list, with dates, of any modifications made, initialled
by the person responsible.
<li>A double row of asterisks down the left hand side to distinguish
the block comment from code.
</ul>

It may be useful to divide the file into sections using form feeds and
section titles centered and double underlined, for example:
<xmp>
			Local Coding Macros
			===================
</xmp>
For each function, a function heading should contain:
<ul>
<li>Before any procedure, a form feed character OR (if the function will
fit on the rest of a 66 line page, or immediately after a section heading),
two blank lines.
<li>An english title for the function (not its name) underlined with dashes
and indented.
<li>On the right, ending in column 80, the function name. This makes it
easier to find when flicking through a listing.
<li>After the title, an abstract of the procedure.
<li>A list of preconditions which MUST be true on entry into the function.
<li>A list of conditions which are true on exit. This includes any change
to global data or system state, anything corrupted.
<li>Optionally, a list of parameters, if these are not clear from comments
against the parameter declarations and in the pre- and post-condistions.
<li>A history list, with dates, of any modifications made specifically
to this function and not recorded under a general heading for the module.
<li>A declaration of the function BOTH in ANSII C AND in Kernighan and Ritchie
C. Use the __STDC__ preprocessor token to select one or the other.
<li>A double row of asterisks down the left hand side to distinguish
any block comment from code.
</ul>
Below is a small example.
<box>
<listing>
<Form Feed>
/*	Allocate an RPC message				               rpc_new()
**	-----------------------
**
**  This procedure must be fast, as it is used in many places in the rpcrts.c
**  and ts.c modules.  For this reason, it keeps a list of allocated messages,
**  and does not return them to the heap when they have been used.
**
**  On entry,
**	p	The address of the pointer to be filled in
**	bytes	Irrelevant. In principle this was to be allowed for a
**		message size required, if we had variable sized messages.
**  On exit,
**	*p	is either the address of a message, or zero if there is
**		not enough memory.
*/

/*ARGSUSED*/
#ifdef __STDC__
void rpc_new(
   rpc_message_pointer  *p,		/* -> Pointer to be returned	*/
   rpc_integer		bytes)		/* Size: ignored		*/
#else
void rpc_new(p, bytes)
   rpc_message_pointer  *p;
   rpc_integer		bytes;
#endif
{
    if (free_messages == NULL) {
        *p = (rpc_message_pointer)malloc((unsigned)(sizeof(*(*p))));
    } else {
        *p = free_messages;
        free_messages = (*p)->m_next;
    } /* if */
} /* rpc_new */
</listing>
</box>
Within a function,
<ul>
<li>Indent by four columns at each open brace "{".
<li>Please put the openning brace at the end of the same line as
the if, while, etc.
<li>Line up the closing brace with the first character of the opening "if",
"while", etc.: <hp2>not</hp2> with the enclosed code.
<li>Comment the closing brace with the function name (for a function block),
or with "if" or "while" etc. For large blocks, give some clue
as to which "if", for example "/* if bad status */".
When closing an "else" block, give the correct sense of the comment
to the closing brace, e.g. "/* else good status */".
<li>Leave spaces on either side of = and ==, but not inside round brackets.
<li>Do not perform an assignment in the outer level of a conditional
expression. For example,
<xmp>
	if (pointer = mallloc(sizeof(*pointer)) {
		printf("malloc was successful\n");
	}
</xmp>
is confusing to most people and gives a warning with some compilers.
</ul>
<h3>Use of the preprocessor.
<i1>Preprocessor, use of
<ul>
<li>All macros should be entirely in upper case unless they completely
emulate functions, in which case they may be entirely in lower case.
<li>When an include file refers to a data type which is a pointer to
a structure defined in another module, then a preprocessor token
may be used to allow either the relevant include file to be included,
or for a dummy declaration (int *) to be made. This allows
one to hide the structures when knowledge of them is not required.
It allows one to restrict the effort involved in a remake, and to restrict the
number of include files distributed.
<li>Tokens beginning and ending with two underscores are reserved for
interaction with the compiler. Do not use them for anything else.
(Apart from causing confusion to people, this upsets the OS9 compiler)
</ul>
<h3>System-specific code
Any <hp2>system-specific code</hp2>
may, if large, be consigned to a completely
separate module, or may be controlled using macros.
In the latter case, no reference to specific machines or operating systems
should be put in the text, only in a file called syspec.h. This file
defines, for a given system, some alternative macros and flags.
For example, code depending on the byte order might use a LSBFIRST flag
which in turn is set by syspec.h.
The syspec.h file is in fact a set of conditional inclusions of
different system_specific files, one for each system.
It uses the compiler predefined tokes to select the relevant file.
In this way, extra machines can sometimes be added without touching
the distributed files, except adding a line to syspec.h.

Macros which may be altered by syspec.h should be defined conditionally
on their not having been defined before.
<xmp>
#ifndef PANIC
#define PANIC fprint("Panic!\n"); exit(-999);
#endif
</xmp>
<h3>Utilities
Compiling a module with an ANSI C compiler is a useful way
of checking for nonportable aspects and for bugs, especially
if you have included ANSI C headings.
The TurboC compiler on MSDOS is fairly fussy, for example.

`lint' is a unix tool for checking C files for consistency.
It is not perfect, and code does not have to be completely "lint free"
but it is is useful, and code should run though line with as few warnings
as possible.
Note the use of a lint specila comment /*ARGSUSED*/ to prevent lint complaining
that a deliberately unused argument had not been referenced.
There are a few other similar special comments (See `man lint' on unix.)

There are various "prettifiers" which aim to format C files. `indent' from Gnu
and unix `cb' are two. These do not do a good enough job: they tend to make a
mess of comments, for example. Therefore, do not assume you can clean up your
code automatically afterwards!
<h2>Software distribution

<i1>Distribution, software
<i1>Versions, software
This scheme roughly follows the DD/OC MODEL scheme, leaving a little
more flexibility.

A version directory is one whose name is vx_y where the version
is version x.y. (eg [RPC.UTIL.V2_1]).

<ol>
<li>Under a version directory the subdirectory distribution_files
should contain all the files to be distributed with the release.

<li>Under the version directory the subdirectory MASTERS
(eg [RPC.V2_1.MASTERS]) should contain the master sources
of any command files, release notes etc applicable only to that release.

<li>In the version directory, the BUILD.COM command file must
<ol>
<li>Copy all files from the development directories and the
masters subdirectory into the distribution_files subdirectory
<li>Create a VMS instalation kit (.A, if appropriate), or otherwise a backup
(.BCK) file containing all the distributed files.
</ol>
<li>The file SETUP.COM (for VMS software) should be released with the
<i1>SETUP.COM
files and perform any necessary logon time set&hyphen.up for a user to
be able to use the software. The source for this will normally
be in the masters subdirectory.
</ol>
<xmp>
..tb 9 17 25 33 41 49 57 65 73
                [...V2_1]
                    |   build.com
                    |
         ___________|___________________
        |                               |
    MASTERS                     RELEASE_FILES
        setup.com                       all files included in release
        xxvvv_release_notes
</xmp>
Note: One needs priviledge to build a VMS/INSTAL set,
<i1>VMS/INSTAL
so in practice this cannot be done on VXCERN:
Nor can an INSTAL procedure be done by a nonpriviledged user
who wants to pick up a version of the software on another machine.
For systems which do not require priviledge to set up, this simpler
SETUP.COM system is used instead. 
<appendix>
<h1>Differences from "hermes"

<i1>hermes
This section notes some of the differences between the DD/OC RPC system
and the "hermes" system from Eric Cooper at Berkley, adapted
by Kris Kostro at CERN<BIBREF REFID=HERMES>.

The two systems are similar in that they are both remote procedure call
systems using an RPC compiler for stub generation.

<H2>Parameter passing

Whereas the hermes project
<BIBREF refid=hermes>
is designed to interface to C programs, in DD/OC we (and our users) require
remote C, Pascal and
FORTRAN and PILS programs.
The calling conventions therefore differ.

The "hermes" system would only allow data to be returned as a function
value, wheras we must allow all combinations of IN, INOUT and OUT
parameters.

<H2>Linking

There is no provision in the hermes system for the linking
of distributed system to take place transparently.
This is only provided, in the DD/OC system, under VMS.

<H2>Description file

The hermes system uses a special "MESA" language, rather than Ada.

<H2>Message Format

The message format in hermes has been adapted from the Courier standard
in various ways.

<H2>Compiler generation

The hermes compiler ("rig") is built using the UNIX tools "lex" and "yacc",
and produces output in C.
This RPC compiler is written in Pascal, and produces Pascal, FORTRAN,
PILS or C output.

<H2>Protocol

The hermes system's built&hyphen.in protocol
uses a concept of a transient connection, which lapses on
timeout.  The same connection is used by the transaction protocol and by
the rpc system.
In the OC system, a transport connection is established at program
initialisation time, and kept whilst the program is running.

<h1>Comparison with Sun XDR RPC

<i1>Sun XDR
<i1>XDR, Sun
<ul>
<li>
Sun RPC supports a variety of call&hyphen.time authentication schemes.
These are relative heavy schemes, based on unix principles.
<li>
The XDR language used to describe operations and data types
is in the C idiom.
<li>
There is a run time check on the version of RPC protocol, as well
as on the versions of the stubs.  This is a good idea.
<li>
The marshalling routines will handle pointers, but not NIL ones.
I guess this means they will recreate linked data structures.
(but why not NIL pointers?)
<li>
Following the C idiom, all parameters are considered IN parameters,
and the function result may be a complex structure.
<li>
The marshalling format is different: in particular, everything
is a multiple of 32 bits. (With us it is 8 or 16, depending on the
ALIGN option).
<li>
The (un)marshalling routines put (take) data directly
to (from) a serial data stream, rather than via a buffer.
This is an interesting difference of philosophy: it relies on the Unix
concept of bytes stream oriented communication.
I feel that in our environments, this would be too slow, as there
would be too many calls to the communications services.
However, this approach has the advantage of having no limitations
imposed by the buffering available within the RPC system.
</ul>

The conclusion is that Sun RPC was not optimised for very fast calls
at run time, as there is quite a large call time overhead.

Similarities between the two systems include

<ul>
<li>The basic RPC concept.
<li>The basic byte ordering is the same, and constant.
</ul>

<h1>Glossary
In response to a specific request, here is a list of some of the terms
used in this document, with short definitions.
<dl>
<dt>Application
<i1>Application, definition
<dd>Code which is directly, rather than indirectly, useful.
<dt>AST
<i1>AST, definition
<dd>Originally, VMS term "Asynchronous System Trap".
A user procedure which is executed in the context of a user process,
but for which the original activity of that process was interrupted.
A call to an AST is forced by the system, just as though the user
code had executed a procedure call.
See an application note by TBL, or VAX/VMS documentation.
<dt>Callback
<i1>Callback, definition
<dd>Within one remote procedure, the calling of another remote
procedure which is executed by the process which called
the first remote procedure. See the user manual.
<dt>Client
<i1>Client, definition
<dd>That which uses a service provided by another.
In RPC, that which calls a procedure executed by another.
See user manual.
A process is a client in relation to a particular remote procedure:
the same process may be server of some procedures and client of others.
<dt>Client Stub
<dd>A dummy module emulating a remote procedure on a local machine.
See user manual.
<dt>Connection
<i1>Connection, definition
<dd>An agreement between communicating transport services which allows orderly
and reliable communication.
For the transport service, this involves keeping a certain amount of
information about messages sent or received.
<hp1>Connection identifier:</hp1> A number by which the user of the Transport
Service
refers to a particular connection.
<hp1>Connection oriented:</hp1> Of a transport service, providing
orderly reliable communication by the use of connections.
<hp1>Connectionless:</hp1> Not connection oriented.
A connectionless communications service normally allows the unrelaible
exchange of messages
<dt>Datagram
<i1>Datagram, definition
<dd>A message sent over a connectionless communication service.
<dt>Disjoint
<dd>Not overlapping. <hp1>Disjoint address spaces:</hp1>
address space not accessible from the same process,
and therefore not allowing communication using shared memory.
<dt>Handle
<i1>Handle, definition
<dd>A reference number by which a Client Stub refers to its
Peer Server Stub.
<dt>INCLUDE
<i1>INCLUDE preprocessor
<dd>A source code preprocesor developed as part of the CERN
cross&hyphen.software suite.
This allows conditional selection of code sections,
and the embedding of code from other files.
It is used in the generation of the RPC software for specific target
environments.
<dt>Link
<dd>A term sometimes used for "Connection".
<dt>Marshalling
<i1>Marshalling, definition.
<dd>The action of taking data from the application data area, converting
it into a suitable format, and putting it into a message ready for
transmission.
<dt>Medium
<i1>Medium, definition
<dd>That through which things travel.
In this document, the method of communicaication including
all the services provided by the operating system.
<dt>Message
<i1>Message, definition
<dd>1. A logical unit of communication, containing an ordered set of bytes,
and having a defined length.
2. A data structure used within the RPC system to store such a message
and also some associated information.
<dt>Package
<i1>Package, definition
<dd>An ordered set of procedures and functions.
A package is defined by one definition file.
A package is the unit of RPC compilation,
and a unit of addressing at run time.
See the user manual.
<dt>Packet
<i1>Packet, definition
<dd>A unit of communication over a network. A term generally used at the
link layer, e.g. for an ethernet packet.
Packets tend to have limited sizes.
<dt>Packing
<dd>A term sometimes used for "Marshalling".
"Marshalling" is preferred because "packing" suggests expressing data in
a more compact format, as in the pascal "packed array".
<dt>Peer
<i1>Peer, definition
<dd>Another on an equal standing. Partner on the same level.
<hp1>To be tried by one's peers:</hp1> to be tried by those
of equal social status.
<hp1>Peer address:</hp1> The address of the equivalent entity with which one
is communicating.
<dt>Process
<dd>A serial thread of execution by a computer.
<dt>Remote
<dd>At a distance.
In RPC, strictly: in a different process with a disjoint address space.
Alternatively, on a different processor.
<dt>Run Time Support
<dd>A body of software providing to the Stubs those services
which they require.
<dt>Segmentation
<i1>Segmentation, definition
<dd>The process or dividing large messages into small packets,
and of reassembling them into the original messages.
<dt>Server
<i1>Server, definition
<dd>That which provides a service to another.
In RPC, that process which executes a procedure called by another process.
<dt>Service
<dd>Useful function.
<dt>Server Stub
<dd>A dummy module emulating a remote calling program on a local machine.
See user manual.
<dt>Socket
<i1>Socket, definition
<dd>A channel of communication
provided by the RPC transport service.
A socket number is generated by the transport layer when a socket
is opened by calling TS_Open.
The number is quoted for all communications functions referring to
that channel.
Internally to the TS, the socket number
is typically a pointer to a data structure describing
the channel. (The term 'channel' is used here only in explanation:
it is not used in the RPC system to refer to a socket).
(Berkley unix has a 'socket library' with a similar concept.)
<dt>Stub
<i1>Stub, definition
<dd>A dummy module emulating a remote module on a local machine.
See user manual.
<dt>Task
<dd>Common term for Process, e.g. in RMS68K.
The term "process" is preferred in this document,
but terms monotask and multitask are current.
<dt>TS
<dd>1. Transport Service.
2. The module of this RPC system which provides a transport service to
the RPCRTS module. (See the diagram at the start of Chapter 3)
<dt>TSAP
<i1>TSAP, definition.
<dd>Transport Service Access Point. An ISO term, denoting a thing to which
a connection may be made. The name of the TSAP to which you want to connect
is what you give to an ISO transport service when asking for a new connection.
The DECNET equivalent term is "Decnet object".
<dt>Transport Service
<i1>Transport Service, definition
<dd>The function of reliable communication of information between
different processes in disjoint address spaces.
<dt>Unit
<dd>Old name for a package. See "package".
</dl>
<backm>
<bl compact=1>
<BIB ID=COURIER>
"Courier: The Remote Procedure Call Protocol", XEROX Corporation,
XSIS 038112, December 1981.
<BIB ID=X409>
CCITT "Red Book" Volume VIII, Recomendation X.409:
"Message Handling Systems: Presentation Transfer Syntax and Notation"
<BIB ID=HERMES>
K. Kostro, CERN/DD/SW,
"Portable Communications with Remote Procedure Call Protocols",
14 November 1985
<BIB ID=RPCUSER>
T. Berners&hyphen.Lee, A. Pastore DD/OC, "RPC User Manual", August 86
<bib id=REALS>
H. von der Schmidt, "32&hyphen.bit Floating Point Formats possibly used in OPAL
and conversion between them"  HvdS/OPAL 13 May 86
<BIB ID=ASN1>
ISO 8824, "Specification of Abstract Syntax Notation One (ASN/1)",
International Standards Organistion.
<BIB ID=ecma127>
European Computer Manufacturers Association,
standard ECMA&hyphen.127
"RPC: Basic Remote Procedure Call using ISO Remote Operations"
UDC 681.327.8.04/.06
ISO/IEC DIS 10148
<BIB ID=iso8649>
ISO 8649, <hp1>Information Processing Systems - Open Systems Interconnection -
Association Control Service Element,</hp1>
International Standards Organistion.
<BIB ID=iso90721>
ISO 9072/1, <hp1>Remote Operations (1): Model, Notation & Service Definition</hp1>
International Standards Organistion.
<BIB ID=iso90722>
ISO 9072/2, <hp1>Remote Operations (2): Protocol Specification,</hp1>
International Standards Organistion.
<BIB ID=NCSRM>
Apollo Computer,
<hp1>Network Computing System Reference Manual</hp1>,
Order number 010200
<BIB ID=NCAPS>
L. Zahn, Apollo Computer,
<hp1>Network Computing Architecture Protocol Specifications</hp1>,
Order number 010201

To find documents, people, programs, etc., related to the DD/OC RPC project
under VMS, type
<fn>
Avaliable on VXCERN.  To use on, put the line
..br
        $ enq*uire:==@disk$d1[timbl.enq]enquire
into your login.com file.
</fn>
"ENQUIRE RPC".
</bl>
<index ix=1>Index
</backm>
