System.Interop “how to” for Smart Mobile Studio
Smart Mobile Studio and the dialect “Smart Pascal” has roots in the Object Pascal community. The compiler and toolkit is written purely using FreePascal and Delphi, and naturally the run-time library for Smart Pascal is heavily influenced by both.
What is less known is that Smart Mobile Studio in general, both the IDE and the RTL architecture, is equally inspired by the Mono project. The entire layout of the IDE is almost identical to Mono-Develop as it existed 4 years ago, while the RTL and application framework borrows from MonoTouch by exposing central constructs (like TApplication) purely through partial classes. These are language features which does not exist in FreeePascal or Embarcadero Delphi.
The unit Interop.pas which is a part of the system namespace (system.interop) is inspired by the mono .NET equivalent. Under Mono the various memory and type transformation classes are spread out over many different locations. Under Smart Mobile Studio we have decided to collect and implement these .NET features in a single unit. Namely system.interop (short for: System interoperability).
Memory management under HTML5
/* Allocate memory buffer, 1024 bytes */ var memBuffer = new ArrayBuffer(1024); /* Allocate R/W access through a typed array */ var memAccess = new UInt8Array( memBuffer );
Needless to say, working with memory like this quickly becomes tiredsome, error prone and messy. Especially when coming from a FreePascal or Delphi background, where a simple call to AllocMem() is enough to reserve memory on the heap.
Memory Management under Smart Mobile Studio
The Interop unit introduces 3 different means of working directly with memory, there are:
- Memory Buffers
- Marshaled classical memory access
Marshaled memory access
Such a reference object contains two pieces of information:
- A reference (handle) to the associated JArrayBuffer
- An entrypoint (offset) into the arraybuffer
For instance, under classical object pascal the following code is quite common:
// pointer type containing the address var mBuffer: PByte; // Allocate memory mBuffer:=AllocMem(1024); try // Work with memory here finally // Release memory freemem(mBuffer); end;
Using marshaled objects Smart Mobile Studio is capable of delivering the exact same:
// pointer type containing the address var mBuffer: TAddress; // Allocate memory mBuffer:=TMarshal.allocMem(1024); try //work with memory here finally // Release memory TMarshal.freemem(mBuffer); end;
Marshaled references also have the benefit of helper functions. So advancing further into an allocated segment is done via the TAddress.Addr() function.
// pointer type containing the address var mBuffer: TAddress; // Allocate memory mBuffer:=TMarshal.allocMem(1024); try mBuffer:=mBuffer.Addr(2); //Advance reference by 2 bytes finally // Release memory TMarshal.freemem(mBuffer); end;
var mBuffer: TAddress := TMarshal.allocMem(1024).addr(2);
While marshaled references and memory allocations are essential for legacy compatability, porting code from freepascal or Delphi to HTML5 – being able to directly interact with a memory segment is vital for any modern platform.
Special attention should be made to the class-procedure Move() and Fill() which are equal to the classical FreePascal and Delphi variations. Unlike their classical counterparts these methods accepts TMemoryHandle rather than Pointer or marshaled TAddress objects.
Using Move with Marshaled Addresses
The TAddress class exposes a property named “Segment” of type TMemoryHandle. This can be used with Move() and Fill() as demonstrated below:
var src: TAddress; dst: TAddress; TMemory.Move( src.segment, // source memory src.entrypoint, // source offset dst.segment, // target memory dst.Entrypoint, // target offset 500 //bytes to move ); TMemory.Fill( src.segment, src.entryPoint, 500, TDatatype.CharToByte("Z") );
It must be noted that the above code is just for explanation. Both Move() and FillChar() are implemented in the TMarshal class. The above demonstration is just to provide reference material on their use and how marshaled memory works.
Using TDatatype is very easy:
var x: Integer; mData: TByteArray; // convert int32 value $1200 into 4 bytes mData := TDatatype.Int32ToBytes($1200); // write bytes to the console for x:=0 to mData.length-1 do writeln(mData[x]); // Alter first byte just for fun mData:=$FF; // Write modified integer value to the console Writeln( TDataType.BytesToInt32(mData) );
The following methods are provided by TDatatype for conversion of datatypes:
TDatatype = Class public class property Bits:TBitAccess; class function Int16ToBytes(Value:Integer):TByteArray; class function Int32ToBytes(Value:Integer):TByteArray; class function Float64ToBytes(Value:Float64):TByteArray; class function Float32ToBytes(Value:Float32):TByteArray; class function StringToBytes(Value:String):TByteArray; class function BooleanToBytes(Value:Boolean):TByteArray; class function BytesToInt16(const Data:TByteArray):Integer; class function BytesToInt32(const Data:TByteArray):Integer; class function BytesToFloat32(Const Data:TByteArray):Float; class function BytesToFloat64(Const Data:TByteArray):Float; class function BytesToString(const Data:TByteArray):String; class function BytesToBoolean(const data:TByteArray):Boolean; class function StrToBase64(Value:String):String; class function Base64ToStr(Value:String):String; class function ByteToChar(Const Value:Byte):String; class function CharToByte(const Value:String):Byte; class function StrToTypedArray(value:String):TJSByteClass; class function TypedArrayToStr(const value:TJSByteClass):String; end;
Working with bits
The TDatatype class exposes a class-property named TBitAccess, which contains only class methods for manipulating bits. As of writing the following methods exists:
TBitAccess = Class public class function Get(Const index:Integer;Const Value:Byte):Boolean; class function &Set(Const Index:Integer;Const Value:Byte; const Data:Boolean):Byte; class function Count(Const Value:Byte):Integer; class function ToByteString(const Value:Byte):String; class function FromByteString(const Value:String):Byte; class function Dismantle(const Value:Byte):TByteArray; class function Assemble(const Value:TByteArray):Byte; end;
Note: TMemory also exposes getBit and setBit methods for the whole allocated memory. This allows you to use a whole segment as a bit-buffer. This is often used by compression, sound generators and classes which communicates with hardware (see Smart Support for hardware platforms).
Working with Streams
While marshaled memory allocations and TMemory represents fast and direct access to raw data, RTL software typically base themselves on streams. In essence Streams and buffers represent the same thing, except that a stream maintains a cursor which is automatically updated as you advance through the content.
Using streams is done much like under FreePascal and Delphi:
var mStream: TMemoryStream; mStream:=TMemoryStream.Create; try // use stream here finally mStream.free; end;
Reading and writing to streams can either be done directly using TStream.Write() or TStream.Read(), both these methods accepts an array of byte (TByteArray). This makes streams slower to use than TMemory or TMarshal operations.
var mStream: TMemoryStream; mWriter: TStreamWriter; mStream:=TMemoryStream.Create; try mWriter:=mStream.createWriter; mWriter.WriteInt32($1200); mWriter.WriteString("This is how you write to a stream!"); mWriter.WriteFloat64(139.68504); finally mStream.free; end;
Reading is done via TStreamReader, which you can either create manually or obtain via the TStream.CreateReader() method.
In the first test, which writes 100.000 records into a pre-allocated TMemory instance, we get the following results:
mMake.Allocate(2100021); mStart:=Now; mChunk:=0; for x:=0 to 100000 do begin mMake.Write(mChunk + 0,$AAAA0000); mMake.WriteFloat32(mChunk + 4,19.24); mMake.WriteFloat64(mChunk + 8,24.18); mMake.Write(mChunk + 16,false); mMake.write(mChunk + 17,$0000AAAA); inc(mChunk, 21); end; mStop:=Now;
Time to completion: 0 seconds, 46 milliseconds
In the second test we omit pre-allocation, which will force TMemory to grow and re-allocate it’s cache quite often, depending on the size of the cache (4096 bytes in this case):
Time to completion: 0 seconds, 445 milliseconds
In the third test we do the exact same, but this time using streams and streamwriter. To fully understand what is happening here, for each write, consider the following:
- StreamWriter calls TDatatype to convert the intrinsic value into a TByteArray
- StreamWriter calls TStream to write the data
- TStream maintains a TMemory class, and calls the rather slow Write() operation
- TStream’s TMemory instance has no pre-allocation, and as such re-allocation will occur often
Time to completion: 0 seconds, 519 milliseconds
As of writing my code is roughly two times faster than Perl and Python, it’s also faster than any native Embarcadero Delphi version except Delphi XE7. Only FreePascal and C# Mono compiled to machine-code using LLVM optimization is capable of performance like this. Which is to some degree embarrassing and humiliating for Delphi, a product costing 100 times more than Smart Mobile Studio.
On the positive side, it spells a bright future for Smart Mobile Studio, especially when interacting with nodeJS and hardware, which require powerful and fast code.