Archive

Archive for the ‘Delphi’ Category

30% discount on all RemObjects products!

July 8, 2019 Leave a comment

This is brilliant. RemObjects is giving a whopping 30% discount on all products!

This means you can now pick up RemObjects Remoting Framework, Data Abstract, Hydra or the Elements compiler toolchain – with a massive 30% saving!

These are battle-hardened, enterprise level solutions that have been polished over years and they are in constant development. Each solution integrates seamlessly into Embarcadero Delphi and provides a smooth path to delivering quality products in days rather than weeks.

But you better hurry because it’s only valid for one week (!)

Use the coupon code: “DelphiDeveloper”

66825092_10156336639680906_8015817715019153408_o

Use the Delphi Developer coupon to get 30% discount – click here

 

A Delphi propertybag

July 7, 2019 8 comments

A long, long time ago, way back in the previous century, I often had to adjust a Visual Basic project my company maintained. Going from object-pascal to VB was more than a little debilitating; Visual Basic was not a compiled language like Delphi is, and it lacked more or less every feature you needed to produce good software.

source

I could probably make a VB clone using Delphi pretty easily. But I think the world has experienced enough suffering, no need to add more evil to the universe

Having said that, I have always been a huge fan of Basic (it was my first language after all, it’s what schools taught in the 70s and 80s). I think it was a terrible mistake for Microsoft to retire Basic as a language, because it’s a great way to teach kids the fundamentals of programming.

Visual Basic is still there though, available for the .Net framework, but to call it Basic is an insult of the likes of GFA Basic, Amos Basic and Blitz Basic; the mighty compilers of the past. If you enjoyed basic before Microsoft pushed out the monstrosity that is Visual Basic, then perhaps swing by GitHub and pick up a copy of BlitzBasic?  BlitzBasic is a completely different beast. It compiles to machine-code, allows inline assembly, and has been wildly popular for game developers over the years.

A property bag

The only feature that I found somewhat useful in Visual Basic, was an object called a propertybag. It’s just a fancy name for a dictionary, but it had a couple of redeeming factors beyond lookup ability. Like being able to load name-value-pairs from a string, recognizing datatypes and exposing type-aware read/write methods. Nothing fancy but handy when dealing with database connection-strings, shell parameters and the like.

So you could feed it strings like this:

first=12;second=hello there;third=3.14

And the class would parse out the names and values, stuff it in a dictionary, and you could easily extract the data you needed. Nothing fancy, but handy on rare occasions.

A Delphi version

Im mostly porting code from Delphi to Oxygene these days, but here is my Delphi implementation of the propertybag object. Please note that I haven’t bothered to implement the propertybag available in .Net. The Delphi version below is based on the Visual Basic 6 version, with some dependency injection thrown in for good measure.

unit fslib.params;

interface

{.$DEFINE SUPPORT_URI_ENCODING}

uses
  System.SysUtils,
  System.Classes,
  Generics.Collections;

type

  (* Exceptions *)
  EPropertybag           = class(exception);
  EPropertybagReadError  = class(EPropertybag);
  EPropertybagWriteError = class(EPropertybag);
  EPropertybagParseError = class(EPropertybag);

  (* Datatypes *)
  TPropertyBagDictionary = TDictionary ;

  IPropertyElement = interface
    ['{C6C937DF-50FA-4984-BA6F-EBB0B367D3F3}']
    function  GetAsInt: integer;
    procedure SetAsInt(const Value: integer);

    function  GetAsString: string;
    procedure SetAsString(const Value: string);

    function  GetAsBool: boolean;
    procedure SetAsBool(const Value: boolean);

    function  GetAsFloat: double;
    procedure SetAsFloat(const Value: double);

    function  GetEmpty: boolean;

    property Empty: boolean read GetEmpty;
    property AsFloat: double read GetAsFloat write SetAsFloat;
    property AsBoolean: boolean read GetAsBool write SetAsBool;
    property AsInteger: integer read GetAsInt write SetAsInt;
    property AsString: string read GetAsString write SetAsString;
  end;

  TPropertyBag = Class(TInterfacedObject)
  strict private
    FLUT:       TPropertyBagDictionary;
  strict protected
    procedure   Parse(NameValuePairs: string);
  public
    function    Read(Name: string): IPropertyElement;
    function    Write(Name: string; Value: string): IPropertyElement;

    procedure   SaveToStream(const Stream: TStream);
    procedure   LoadFromStream(const Stream: TStream);
    function    ToString: string; override;
    procedure   Clear; virtual;

    constructor Create(NameValuePairs: string); virtual;
    destructor  Destroy; override;
  end;

implementation

{$IFDEF SUPPORT_URI_ENCODING}
uses
  system.NetEncoding;
{$ENDIF}

const
  cnt_err_sourceparameters_parse =
  'Failed to parse input, invalid or damaged text error [%s]';

  cnt_err_sourceparameters_write_id =
  'Write failed, invalid or empty identifier error';

  cnt_err_sourceparameters_read_id =
  'Read failed, invalid or empty identifier error';

type

  TPropertyElement = class(TInterfacedObject, IPropertyElement)
  strict private
    FName:      string;
    FData:      string;
    FStorage:   TPropertyBagDictionary;
  strict protected
    function    GetEmpty: boolean; inline;

    function    GetAsInt: integer; inline;
    procedure   SetAsInt(const Value: integer); inline;

    function    GetAsString: string; inline;
    procedure   SetAsString(const Value: string); inline;

    function    GetAsBool: boolean; inline;
    procedure   SetAsBool(const Value: boolean); inline;

    function    GetAsFloat: double; inline;
    procedure   SetAsFloat(const Value: double); inline;

  public
    property    AsFloat: double read GetAsFloat write SetAsFloat;
    property    AsBoolean: boolean read GetAsBool write SetAsBool;
    property    AsInteger: integer read GetAsInt write SetAsInt;
    property    AsString: string read GetAsString write SetAsString;
    property    Empty: boolean read GetEmpty;

    constructor Create(const Storage: TPropertyBagDictionary; Name: string; Data: string); overload; virtual;
    constructor Create(Data: string); overload; virtual;
  end;

//#############################################################################
// TPropertyElement
//#############################################################################

constructor TPropertyElement.Create(Data: string);
begin
  inherited Create;
  FData := Data.Trim();
end;

constructor TPropertyElement.Create(const Storage: TPropertyBagDictionary;
  Name: string; Data: string);
begin
  inherited Create;
  FStorage := Storage;
  FName := Name.Trim().ToLower();
  FData := Data.Trim();
end;

function TPropertyElement.GetEmpty: boolean;
begin
  result := FData.Length < 1;
end;

function TPropertyElement.GetAsString: string;
begin
  result := FData;
end;

procedure TPropertyElement.SetAsString(const Value: string);
begin
  if Value  FData then
  begin
    FData := Value;
    if FName.Length > 0 then
    begin
      if FStorage  nil then
        FStorage.AddOrSetValue(FName, Value);
    end;
  end;
end;

function TPropertyElement.GetAsBool: boolean;
begin
  TryStrToBool(FData, result);
end;

procedure TPropertyElement.SetAsBool(const Value: boolean);
begin
  FData := BoolToStr(Value, true);

  if FName.Length > 0 then
  begin
    if FStorage  nil then
      FStorage.AddOrSetValue(FName, FData);
  end;
end;

function TPropertyElement.GetAsFloat: double;
begin
  TryStrToFloat(FData, result);
end;

procedure TPropertyElement.SetAsFloat(const Value: double);
begin
  FData := FloatToStr(Value);
  if FName.Length > 0 then
  begin
    if FStorage  nil then
      FStorage.AddOrSetValue(FName, FData);
  end;
end;

function TPropertyElement.GetAsInt: integer;
begin
  TryStrToInt(FData, Result);
end;

procedure TPropertyElement.SetAsInt(const Value: integer);
begin
  FData := IntToStr(Value);
  if FName.Length > 0 then
  begin
    if FStorage  nil then
      FStorage.AddOrSetValue(FName, FData);
  end;
end;

//#############################################################################
// TPropertyBag
//#############################################################################

constructor TPropertyBag.Create(NameValuePairs: string);

begin
  inherited Create;
  FLUT := TDictionary.Create();

  NameValuePairs := NameValuePairs.Trim();
  if NameValuePairs.Length > 0 then
    Parse(NameValuePairs);
end;

destructor TPropertyBag.Destroy;
begin
  FLut.Free;
  inherited;
end;

procedure TPropertyBag.Clear;
begin
  FLut.Clear;
end;

procedure TPropertyBag.Parse(NameValuePairs: string);
var
  LList:      TStringList;
  x:          integer;
  LId:        string;
  LValue:     string;
  LOriginal:  string;
  {$IFDEF SUPPORT_URI_ENCODING}
  LPos:       integer;
  {$ENDIF}
begin
  // Reset content
  FLUT.Clear();

  // Make a copy of the original text
  LOriginal := NameValuePairs;

  // Trim and prepare
  NameValuePairs := NameValuePairs.Trim();

  // Anything to work with?
  if NameValuePairs.Length > 0 then
  begin
    {$IFDEF SUPPORT_URI_ENCODING}
    // Check if the data is URL-encoded
    LPos := pos('%', NameValuePairs);
    if  (LPos >= low(NameValuePairs) )
    and (LPos  0 then
    Begin
      (* Populate our lookup table *)
      LList := TStringList.Create;
      try
        LList.Delimiter := ';';
        LList.StrictDelimiter := true;
        LList.DelimitedText := NameValuePairs;

        if LList.Count = 0 then
          raise EPropertybagParseError.CreateFmt(cnt_err_sourceparameters_parse, [LOriginal]);

        try
          for x := 0 to LList.Count-1 do
          begin
            LId := LList.Names[x].Trim().ToLower();
            if (LId.Length > 0) then
            begin
              LValue := LList.ValueFromIndex[x].Trim();
              Write(LId, LValue);
            end;
          end;
        except
          on e: exception do
          raise EPropertybagParseError.CreateFmt(cnt_err_sourceparameters_parse, [LOriginal]);
        end;
      finally
        LList.Free;
      end;
    end;
  end;
end;

function TPropertyBag.ToString: string;
var
  LItem: TPair;
begin
  setlength(result, 0);
  for LItem in FLut do
  begin
    if LItem.Key.Trim().Length > 0 then
    begin
      result := result + Format('%s=%s;', [LItem.Key, LItem.Value]);
    end;
  end;
end;

procedure TPropertyBag.SaveToStream(const Stream: TStream);
var
  LData: TStringStream;
begin
  LData := TStringStream.Create(ToString(), TEncoding.UTF8);
  try
    LData.SaveToStream(Stream);
  finally
    LData.Free;
  end;
end;

procedure TPropertyBag.LoadFromStream(const Stream: TStream);
var
  LData: TStringStream;
begin
  LData := TStringStream.Create('', TEncoding.UTF8);
  try
    LData.LoadFromStream(Stream);
    Parse(LData.DataString);
  finally
    LData.Free;
  end;
end;

function TPropertyBag.Write(Name: string; Value: string): IPropertyElement;
begin
  Name := Name.Trim().ToLower();
  if Name.Length > 0 then
  begin
    if not FLUT.ContainsKey(Name) then
      FLut.Add(Name, Value);

    result := TPropertyElement.Create(FLut, Name, Value) as IPropertyElement;
  end else
  raise EPropertybagWriteError.Create(cnt_err_sourceparameters_write_id);
end;

function TPropertyBag.Read(Name: string): IPropertyElement;
var
  LData:  String;
begin
  Name := Name.Trim().ToLower();
  if Name.Length > 0  then
  begin
    if FLut.TryGetValue(Name, LData) then
      result := TPropertyElement.Create(LData) as IPropertyElement
    else
      raise EPropertybagReadError.Create(cnt_err_sourceparameters_read_id);
  end else
  raise EPropertybagReadError.Create(cnt_err_sourceparameters_read_id);
end;


end.

BTree for Delphi

July 7, 2019 Leave a comment
lookup

Click here to read

A few weeks back I posted an article on RemObjects blog regarding universal code, and how you with a little bit of care can write code that easily compiled with both Oxygene, Delphi and Freepascal. With emphasis on Oxygene.

The example I used was a BTree class that I originally ported from Delphi to Smart Pascal, and then finally to Oxygene to run under WebAssembly.

Long story short I was asked if I could port the code back to Delphi in its more or less universal form. Naturally there are small differences here and there, but nothing special that distinctly separates the Delphi version from Oxygene or Smart Pascal.

Why this version?

If you google BTree and Delphi you will find loads of implementations. They all operate more or less identical, using records and pointers for optimal speed. I decided to base my version on classes for convenience, but it shouldn’t be difficult to revert that to use records if you absolutely need it.

What I like about this BTree implementation is that it’s very functional. Its easy to traverse the nodes using the ForEach() method, you can add items using a number as an identifier, but it also supports string identifiers.

I also changed the typical data reference. The data each node represent is usually a pointer. I changed this to variant to make it more functional.

Well, here is the Delphi version as promised. Happy to help.

unit btree;

interface

uses
  System.Generics.Collections,
  System.Sysutils,
  System.Classes;

type

  // BTree leaf object
  TQTXBTreeNode = class(TObject)
  public
    Identifier: integer;
    Data:       variant;
    Left:       TQTXBTreeNode;
    Right:      TQTXBTreeNode;
  end;

  [Weak]
  TQTXBTreeProcessCB = reference to procedure (const Node: TQTXBTreeNode; var Cancel: boolean);

  EBTreeError = class(Exception);

  TQTXBTree = class(TObject)
  private
    FRoot:    TQTXBTreeNode;
    FCurrent: TQTXBTreeNode;
  protected
    function  GetEmpty: boolean;  virtual;
    function  GetPackedNodes: TList;

  public
    property  Root: TQTXBTreeNode read FRoot;
    property  Empty: boolean read GetEmpty;

    function  Add(const Ident: integer; const Data: variant): TQTXBTreeNode; overload; virtual;
    function  Add(const Ident: string; const Data: variant): TQTXBTreeNode; overload; virtual;

    function  Contains(const Ident: integer): boolean; overload; virtual;
    function  Contains(const Ident: string): boolean; overload; virtual;

    function  Remove(const Ident: integer): boolean; overload; virtual;
    function  Remove(const Ident: string): boolean; overload; virtual;

    function  Read(const Ident: integer): variant; overload; virtual;
    function  Read(const Ident: string): variant; overload; virtual;

    procedure Write(const Ident: string; const NewData: variant); overload; virtual;
    procedure Write(const Ident: integer; const NewData: variant); overload; virtual;

    procedure Clear; overload; virtual;
    procedure Clear(const Process: TQTXBTreeProcessCB); overload; virtual;

    function  ToDataArray: TList;
    function  Count: integer;

    procedure ForEach(const Process: TQTXBTreeProcessCB);

    destructor Destroy; override;
  end;

implementation

//#############################################################################
// TQTXBTree
//#############################################################################

destructor TQTXBTree.Destroy;
begin
  if FRoot  nil then
    Clear();
  inherited;
end;

procedure TQTXBTree.Clear;
var
  lTemp:  TList;
  x:  integer;
begin
  if FRoot  nil then
  begin
    // pack all nodes to a linear list
    lTemp := GetPackedNodes();

    try
      // release each node
      for x := 0 to ltemp.Count-1 do
      begin
        lTemp[x].Free;
      end;
    finally
      // dispose of list
      lTemp.Free;

      // reset pointers
      FCurrent := nil;
      FRoot := nil;
    end;
  end;
end;

procedure TQTXBTree.Clear(const Process: TQTXBTreeProcessCB);
begin
  ForEach(Process);
  Clear();
end;

function TQTXBTree.GetPackedNodes: TList;
var
  LData:  Tlist;
begin
  LData := TList.Create();
  ForEach( procedure (const Node: TQTXBTreeNode; var Cancel: boolean)
  begin
    LData.Add(Node);
    Cancel  := false;
  end);
  result := LData;
end;

function TQTXBTree.GetEmpty: boolean;
begin
  result := FRoot = nil;
end;

function TQTXBTree.Count: integer;
var
  LCount: integer;
begin
  ForEach( procedure (const Node: TQTXBTreeNode; var Cancel: boolean)
    begin
      inc(LCount);
      Cancel  := false;
    end);
  result := LCount;
end;

function TQTXBTree.ToDataArray: TList;
var
  Data: TList;
begin
  Data := TList.Create();

  ForEach( procedure (const Node: TQTXBTreeNode; var Cancel: boolean)
    begin
      Data.add(Node.data);
      Cancel := false;
    end);
  result := data;
end;

function TQTXBTree.Add(const Ident: string; const Data: variant): TQTXBTreeNode;
begin
  result := Add( Ident.GetHashCode(), Data);
end;

function TQTXBTree.Add(const Ident: integer; const Data: variant): TQTXBTreeNode;
var
  lNode:  TQTXBtreeNode;
begin
  LNode := TQTXBTreeNode.Create();
  LNode.Identifier := Ident;
  LNode.Data := data;

  if FRoot = nil then
    FRoot := LNode;

  FCurrent := FRoot;

  while true do
  begin
    if (Ident  FCurrent.Identifier) then
    begin
      if (FCurrent.right = nil) then
      begin
        FCurrent.right := LNode;
        break;
      end else
      FCurrent := FCurrent.right;
    end else
    break;
  end;
  result := LNode;
end;

function TQTXBTree.Read(const Ident: string): variant;
begin
  result := Read( Ident.GetHashCode() );
end;

function TQTXBTree.Read(const Ident: integer): variant;
begin
  FCurrent := FRoot;
  while FCurrent  nil do
  begin
    if (Ident  Fcurrent.Identifier) then
      FCurrent := FCurrent.Right
    else
    begin
      result := FCUrrent.Data;
      break;
    end
  end;
end;

procedure TQTXBTree.Write(const Ident: string; const NewData: variant);
begin
  Write( Ident.GetHashCode(), NewData);
end;

procedure TQTXBTree.Write(const Ident: integer; const NewData: variant);
begin
  FCurrent := FRoot;
  while (FCurrent  nil) do
  begin
    if (Ident  Fcurrent.Identifier) then
      FCurrent := FCurrent.Right
    else
    begin
      FCurrent.Data := NewData;
      break;
    end
  end;
end;

function  TQTXBTree.Contains(const Ident: string): boolean;
begin
  result := Contains( Ident.GetHashCode() );
end;

function TQTXBTree.Contains(const Ident: integer): boolean;
begin
  result := false;
  if FRoot  nil then
  begin
    FCurrent := FRoot;

    while ( (not Result) and (FCurrent  nil) ) do
    begin
      if (Ident  Fcurrent.Identifier) then
        FCurrent := FCurrent.Right
      else
      begin
        Result := true;
        break;
      end
    end;
  end;
end;

function TQTXBTree.Remove(const Ident: string): boolean;
begin
  result := Remove( Ident.GetHashCode() );
end;

function TQTXBTree.Remove(const Ident: integer): boolean;
var
  LFound: boolean;
  LParent: TQTXBTreeNode;
  LReplacement,
  LReplacementParent: TQTXBTreeNode;
  LChildCount: integer;
begin
  FCurrent := FRoot;
  LFound := false;
  LParent := nil;
  LReplacement := nil;
  LReplacementParent := nil;

  while (not LFound) and (FCurrent  nil) do
  begin
    if (Ident  FCurrent.Identifier) then
    begin
      LParent := FCurrent;
      FCurrent := FCurrent.right;
    end else
    LFound := true;

    if LFound then
    begin
      LChildCount := 0;

      if (FCurrent.left  nil) then
        inc(LChildCount);

      if (FCurrent.right  nil) then
        inc(LChildCount);

      if FCurrent = FRoot then
      begin
        case (LChildCOunt) of
        0:  begin
              FRoot := nil;
            end;
        1:  begin
              if FCurrent.right = nil then
                FRoot := FCurrent.left
              else
                FRoot :=FCurrent.Right;
            end;
        2:  begin
              LReplacement := FRoot.left;
              while (LReplacement.right  nil) do
              begin
                LReplacementParent := LReplacement;
                LReplacement := LReplacement.right;
              end;

            if (LReplacementParent  nil) then
            begin
              LReplacementParent.right := LReplacement.Left;
              LReplacement.right := FRoot.Right;
              LReplacement.left := FRoot.left;
            end else
            LReplacement.right := FRoot.right;
          end;
        end;

        FRoot := LReplacement;
      end else
      begin
        case LChildCount of
        0:  if (FCurrent.Identifier < LParent.Identifier) then
            Lparent.left  := nil else
            LParent.right := nil;
        1:  if (FCurrent.Identifier < LParent.Identifier) then
            begin
              if (FCurrent.Left = NIL) then
              LParent.left := FCurrent.Right else
              LParent.Left := FCurrent.Left;
            end else
            begin
              if (FCurrent.Left = NIL) then
              LParent.right := FCurrent.Right else
              LParent.right := FCurrent.Left;
            end;
        2:  begin
              LReplacement := FCurrent.left;
              LReplacementParent := FCurrent;

              while LReplacement.right  nil do
              begin
                LReplacementParent := LReplacement;
                LReplacement := LReplacement.right;
              end;
              LReplacementParent.right := LReplacement.left;

              LReplacement.right := FCurrent.right;
              LReplacement.left := FCurrent.left;

              if (FCurrent.Identifier < LParent.Identifier) then
                LParent.left := LReplacement
              else
                LParent.right := LReplacement;
            end;
          end;
        end;
      end;
  end;

  result := LFound;
end;

procedure TQTXBTree.ForEach(const Process: TQTXBTreeProcessCB);

  function ProcessNode(const Node: TQTXBTreeNode): boolean;
  begin
    if Node  nil then
    begin
      if Node.left  nil then
      begin
        result := ProcessNode(Node.left);
        if result then
          exit;
      end;

      Process(Node, result);
      if result then
        exit;

      if (Node.right  nil) then
      begin
        result := ProcessNode(Node.right);
        if result then
          exit;
      end;
    end;
  end;

begin
  ProcessNode(FRoot);
end;

end.

Calling node.js from Delphi

July 6, 2019 Leave a comment

We got a good question about how to start a node.js program from Delphi on our Facebook group today (third one in a week?). When you have been coding for years you often forget that things like this might not be immediately obvious. Hopefully I can shed some light on the options in this post.

Node or chrome?

nodeJust to be clear: node.js has nothing to do with chrome or chromium embedded. Chrome is a web-browser, a completely visual environment and ecosystem.

Node.js is the complete opposite. It is purely a shell based environment, meaning that it’s designed to run services and servers, with emphasis on the latter.

The only thing node.js and chrome have in common, is that they both use the V8 JavaScript runtime engine to load, JIT compile and execute scripts at high speed. Beyond that, they are utterly alien to each other.

Can node.js be embedded into a Delphi program?

Technically there is nothing stopping a C/C++ developer from compiling the node.js core system as C++ builder compatible .obj files; files that can then be linked into a Delphi application through references. But this also requires a bit of scaffolding, like adding support for malloc_, free_ and a few other procedures – so that your .obj files uses the same memory manager as your Delphi code. But until someone does just that and publish it, im afraid you are stuck with two options:

  • Use a library called Toby, that keeps node.js in a single DLL file. This is the most practical way if you insist on hosting your own version of node.js
  • Add node.js as a prerequisite and give users the option to locate the node.exe in your application’s preferences. This is the way I would go, because you really don’t want to force users to stick with your potentially outdated or buggy build.

So yes, you can use toby and just add the toby dll file to your program folder, but I have to strongly advice against that. There is no point setting yourself up for maintaining a whole separate programming language, just because you want JavaScript support.

“How many in your company can write high quality WebAssembly modules?”

If all you want to do is support JavaScript in your application, then I would much rather install Besen into Delphi. Besen is a JavaScript runtime engine written in Freepascal. It is fully compatible with Delphi, and follows the ECMA standard to the letter. So it is extremely compatible, fast and easy to use.

Like all Delphi components Besen is compiled into your application, so you have no dependencies to worry about.

Starting a node.js script

The easiest way to start a node.js script, is to simply shell-execute out of your Delphi application. This can be done as easily as:

ShellExecute(Handle, 'open', PChar('node.exe'), pchar('script.js'), nil, SW_SHOW);

This is more than enough if you just want to start a service, server or do some work that doesn’t require that you capture the result.

If you need to capture the result, the data that your node.js program emits on stdout, there is a nice component in the Jedi Component Library. Also plenty of examples online on how to do that.

If you need even further communication, you need to look for a shell-execute that support pipes. All node.js programs have something called a message-channel in the Javascript world. In reality though, this is just a named pipe that is automatically created when your script starts (with the same moniker as the PID [process identifier]).

If you opt for the latter you have a direct, full duplex message channel directly into your node.js application. You just have to agree with yourself on a protocol so that your Delphi code understands what node.js is saying, and visa versa.

UDP or TCP

If you don’t want to get your hands dirty with named pipes and rolling your own protocol, you can just use UDP to let your Delphi application communicate with your node.js process. UDP is practically without cost since its fundamental to all networking stacks, and in your case you will be shipping messages purely between processes on localhost. Meaning: packets are never sent on the network, but rather delegated between processes on the same machine.

In that case, I suggest you ship in the port you want your UDP server to listen on, so that your node.js service acts as the server. A simple command-line statement like:

node.exe myservice.js 8090

Inside node.js you can setup an UDP server with very little fuzz:


function setupServer(port) {
  var os = require("os");
  var dgram = require("dgram");
  var socket = dgram.createSocket("udp4");

  var MULTICAST_HOST = "224.0.0.236";
  var BROADCAST_HOST = "255.255.255.255";
  var ALL_PORT = 60540;
  var MULTICAST_TTL = 1; // Local network

  socket.bind(port);
  socket.on('listening', function() {
    socket.setMulticastLoopback(true);
    socket.setMulticastTTL(MULTICAST_TTL);
    socket.addMembership(multicastHost);
    if(broadcast) { socket.setBroadcast(true); }
  });
  socket.on('message', parseMessage);
}

function parseMessage(message, rinfo) {
try {
  var messageObject = JSON.parse(message);
  var eventType = messageObject.eventType;
  } catch(e) {
  }
}

Note: the code above assumes a JSON text message.

You can then use any Delphi UDP client to communicate with your node.js server, Indy is good, Synapse is a good library with less overhead – there are many options here.

Do I have to learn Javascript to use node.js?

If you download DWScript you can hook-up the JS-codegen library (see library folder in the DWScript repository), and use that to compile DWScript (object pascal) to kick-ass Javascript. This is the same compiler that was used in Smart Mobile Studio.

“Adding WebAssembly to your resume is going to be a hell of a lot more valuable in the years to come than C# or Java”

Another alternative is to use Freepascal, they have a pas2js project where you can compile ordinary object-pascal to javascript. Naturally there are a few things to keep in mind, both for DWScript and Freepascal – like avoiding pointers. But clean object pascal compiles just fine.

If JavaScript is not your cup of tea, or you simply don’t have time to learn the delicate nuances between the DOM (document object model, used by browsers) and the 100% package oriented approach deployed by node.js — then you can just straight up to webassembly.

RemObjects Software has a kick-ass webassembly compiler, perfect if you dont have the energy or time to learn JavaScript. As of writing this is the fastest and most powerful toolchain available. And I have tested them all.

WebAssembly, no Javascript needed

RO-Single-Gear-512You might remember Oxygene? It used to be shipped with Delphi as a way to target Microsoft CLR (common language runtime) and the .net framework.

Since then Oxygene and the RemObjects toolchain has evolved dramatically and is now capable of a lot more than CLR support.

  • You can compile to raw, llvm optimized machine code for 8 platforms
  • You can compile to CLR/.Net
  • You can compile to Java bytecodes
  • You can compile to WebAssembly!

WebAssembly is not Javascript, it’s important to underline that. WebAssembly was created especially for developers using traditional languages, so that traditional compilers can emit web friendly, binary code. Unlike Javascript, WebAssembly is a purely binary format. Just like Delphi generates machine-code that is linked into a final executable, WebAssembly is likewise compiled, linked and emitted in binary form.

If that sounds like a sales pitch, it’s not. It’s a matter of practicality.

  • WebAssembly is completely barren out of the box. The runtime environment, be it V8 for the browser or V8 for node.js, gives you nothing out of the box. You don’t even have WriteLn() to emit text.
  • Google expects compiler makers to provide their own RTL functions, from the fundamental to the advanced. The only thing V8 gives you, is a barebone way of referencing objects and functions on the other side, meaning the JS and DOM world. And that’s it.

So the reason i’m talking a lot about Oxygene and RemObjects Elements (Elements is the name of the compiler toolchain RemObjects offers), is because it ships with an RTL. So you are not forced to start on actual, literal assembly level.

studio

If you don’t want to study JavaScript, Oxygene and Elements from RemObjects is the solution

RemObjects also delivers a DelphiVCL compatibility framework. This is a clone of the Delphi VCL / Freepascal LCL. Since WebAssembly is still brand new, work is being done on this framework on a daily basis, with updates being issued all the time.

Note: The Delphi VCL framework is not just for WebAssembly. It represents a unified framework that can work anywhere. So if you switch from WebAssembly to say Android, you get the same result.

The most important part of the above, is actually not the visual stuff. I mean, having HTML5 visual controls is cool – but chances are you want to use a library like Sencha, SwiftUI or jQueryUI to compose your forms right? Which means you just want to interface with the widgets in the DOM to set and get values.

jQuery UI Bootstrap

You probably want to use a fancy UI library, like jQuery UI. This works perfectly with Elements because you can reference the controls from your WebAssembly module. You dont have to create TButton, TListbox etc manually

The more interesting stuff is actually the non-visual code you get access to. Hundreds of familiar classes from the VCL, painstakingly re-created, and usable from any of the 5 languages Elements supports.

You can check it out here: https://github.com/remobjects/DelphiRTL

Skipping JavaScript all together

I dont believe in single languages. Not any more. There was a time when all you needed was Delphi and a diploma and you were set to conquer the world. But those days are long gone, and a programmer needs to be flexible and have a well stocked toolbox.

At least try the alternatives before you settle on a phone

Knowing where you want to be is half the journey

The world really don’t need yet-another-c# developer. There are millions of C# developers in India alone. C# is just “so what?”. Which is also why C# jobs pays less than Delphi or node.js system service jobs.

What you want, is to learn the things others avoid. If JavaScript looks alien and you feel uneasy about the whole thing – that means you are growing as a developer. All new things are learned by venturing outside your comfort zone.

How many in your company can write high quality WebAssembly modules?

How many within one hour driving distance from your office or home are experts at WebAssembly? How many are capable of writing industrial scale, production ready system services for node.js that can scale from a single instance to 1000 instances in a large, clustered cloud environment?

Any idiot can pick up node.js and knock out a service, but with your background from Delphi or C++ builder you have a massive advantage. All those places that can throw an exception that JS devs usually ignore? As a Delphi or Oxygene developer you know better. And when you re-apply that experience under a different language, suddenly you can do stuff others cant. Which makes your skills valuable.

qtx

The Quartex Media Desktop have made even experienced node / web developers gasp. They are not used to writing custom-controls and large-scale systems, which is my advantage

So would you learn JavaScript or just skip to WebAssembly? Honestly? Learn a bit of both. You don’t have to be an expert in JavaScript to compliment WebAssembly. Just get a cheap book, like “Node.js for beginners” and “JavaScript the good parts” ($20 a piece) and that should be more than enough to cover the JS side of things.

Adding WebAssembly to your resume and having the material to prove you know your stuff, is going to be a hell of a lot more valuable in the years to come than C#, Java or Python. THAT I can guarantee you.

And, we have a wicked cool group on Facebook you can join too: Click here to visit RemObjects Developer.

 

Getting into Node.js from Delphi

July 1, 2019 Leave a comment

Delphi is one of the best development toolchains for Windows. I have been an avid fan of Delphi since it was first released, and before that – Turbo Pascal too. Delphi has a healthy following – and despite popular belief, Delphi scores quite well on the Tiobe Index.

As cool and efficient as Delphi might be, there are situations where native code wont work. Or at the very least, be less efficient than the alternatives. Delphi has a broad wingspan, from low-level assembler all the way to classes and generics. But JavaScript and emerging web technology is based on a completely different philosophy, one where native code is regarded as negative since it binds you to hardware.

Getting to grips with the whole JavaScript phenomenon, be it for mobile, embedded or back-end services, can be daunting if all you know is native code. But thankfully there are alternatives that can help you become productive quickly, something I will brush over in this post.

JavaScript without JavaScript

Before we dig into the tools of the trade, I want to cover alternative ways of enjoying the power of node.js and Javascript. Namely by using compilers that can convert code from a traditional language – and emit fully working JavaScript. There are a lot more options than you think:

qtx

Quartex Media Desktop is a complete environment written purely in JavaScript. Both Server, Cluster and front-end is pure JavaScript. A good example of what can be done.

  • Swift compiles for JavaScript, and Apple is doing some amazing things with the new and sexy SwiftUI tookit. If you know your way around Swift, you can compile for Javascript
  • Go can likewise be compiled to JS:
    • RemObjects Elements supports the Go language. Elements can target both native (llvm), .Net, Java and WebAssembly.
    • Go2Js
    • GopherJs
    • TARDISgo
  • C/C++ can be compiled to asm.js courtesy of EmScripten. It uses clang to first compile your code to llvm bitcode, and then it converts that into asm.js. You have probably seen games like Quake run in the browser? That was asm.js, a kind of precursor to WebAssembly.
  • NS Basic compiles for JavaScript, this is a Visual Basic 6 style environment with its own IDE even

For those coming straight from Delphi, there are a couple of options to pick from:

  • Freepascal (pas2js project)
  • DWScript compiles code to JavaScript, this is the same compiler that we used in Smart Pascal earlier
  • Oxygene, the next generation object-pascal from RemObjects compiles to WebAssembly. This is by far the best option of them all.
studio

I strongly urge you to have a look at Elements, here running in Visual Studio

JavaScript, Asm.js or WebAssembly?

Asm.js is by far the most misunderstood technology in the JavaScript ecosystem, so let me just cover that before we move on:

A few years back JavaScript gained support for memory buffers and typed arrays. This might not sound very exciting, but in terms of speed – the difference is tremendous. The default variable type in JavaScript is what Delphi developers know as Variant. It assumes the datatype of the values you assign to it. Needless to say, there is a lot of overhead when working with variants – so JavaScript suddenly getting proper typed arrays was a huge deal.

It was then discovered that JavaScript could manipulate these arrays and buffers at high speed, providing it only used a subset of the language. A subset that the JavaScript runtime could JIT compile more easily (turn into machine-code).

So what the EmScripten team did was to implement a bytecode based virtual-machine in Javascript, and then they compile C/C++ to bytecodes. I know, it’s a huge project, but the results speak for themselves — before WebAssembly, this was as fast as it got with JavaScript.

WebAssembly

WebAssembly is different from both vanilla JavaScript and Asm.js. First of all, it’s executed at high speed by the browser itself. Not like asm.js where these bytecodes were executed by JavaScript code.

water

Water is a fast, slick and platform independent IDE for Elements. The same IDE for OS X is called Fire. You can use RemObjects Elements from either Visual Studio or Water

Secondly, WebAssembly is completely JIT compiled by the browser or node.js when loading. It’s not like Asm.js where some parts are compiled, others are interpreted. WebAssembly runs at full speed and have nothing to do with traditional JavaScript. It’s actually a completely separate engine.

Out of all the options on the table, WebAssembly is the technology with the best performance.

Kits and strategies

The first thing you need to be clear about, is what you want to work with. The needs and requirements of a game developer will be very different from a system service developer.

Here are a couple of kits to think about:

  • Mobile developer
    • Implement your mobile applications using Oxygene, compiling for WebAssembly (Elements)
    • RemObjects Remoting SDK for client / server communication
    • Use Freepascal for vanilla JavaScript scaffolding when needed
  • Service developer
    • Implement libraries in Oxygene to benefit from the speed of WebAssembly
    • Use RemObjects Data Abstract to make data-access uniform and fast
    • Use Freepascal for boilerplate node.js logic
  • Desktop developer
    • For platform independent desktop applications, WebAssembly is the way to go. You will need some scaffolding (plain Javascript) to communicate with the application host  – but the 99.9% of your code will be better under WebAssembly.
    • Use Cordova / Phonegap to “bundle” your WebAssembly, HTML5 files and CSS styling into a single, final executable.

The most important part to think about when getting into JavaScript, is to look closely at the benefits and limitation of each technology.

WebAssembly is fast, wicked fast, and let’s you write code like you are used to from Delphi. Things like pointers etc are supported in Elements, which means ordinary code that use pointers will port over with ease. You are also not bound on hand-and-feet to a particular framework.

For example, EmScripten for C/C++ have almost nothing in terms of UI functionality. The visual part is a custom build of SDL (simple directmedia layer), which fakes the graphics onto an ordinary HTML5 canvas. This makes EmScripten a good candidate for porting games written in C/C++ to the web — but it’s less than optimal for writing serious applications.

Setting up the common tools

So far we have looked at a couple of alternatives for getting into the wonderful world of JavaScript in lieu of other languages. But what if you just want to get started with the typical tools JS developers use?

vscode

Visual Studio Code is a pretty amazing code-editor

The first “must have” is Visual Studio Code. This is actually a great example of what you can achieve with JavaScript, because the entire editor and program is written in JavaScript. But I want to stress that this editor is THE editor to get. The way you work with files in JS is very different from Delphi, C# and Java. JavaScript projects are often more fragmented, with less code in each file – organized by name.

typescript

TypeScript was invented by Anders Hejlsberg, who also made Delphi and C#

The next “must have” is without a doubt TypeScript. Personally im not too fond of TypeScript, but if ordinary JavaScript makes your head hurt and you want classes and ordinary inheritance, then TypeScript is a step up.

assemblyscriptNext on the list is AssemblyScript. This is a post-processor for TypeScript that converts your code into WebAssembly. It lacks much of the charm and elegance of Oxygene, but I suspect that has to do with old habits. When you have been reading object-pascal for 20 years, you feel more at home there.

nodeYou will also need to install node.js, which is the runtime engine for running JavaScript as services. Node.js is heavily optimized for writing server software, but it’s actually a brilliant way to write services that are multi-platform. Because Node.js delivers the same behavior regardless of underlying operating system.

phonegapAnd finally, since you definitely want to convert your JavaScript and/or WebAssembly into a stand-alone executable: you will need Adobe Phonegap.

Visual Studio

No matter if you want to enter JavaScript via Elements or something else, Visual Studio will save you a lot of time, especially if you plan on targeting Azure or Amazon services. Downloading and installing the community edition is a good idea, and you can use that while exploring your options.

dotnet-visual-studio

When it comes to writing system services, you also want to check out NPM, the node.js package manager. The JavaScript ecosystem is heavily package oriented – and npm gives you some 800.000 packages to play with free of charge.

Just to be clear, npm is a shell command you use to install or remove packages. NPM is also a online repository of said packages, where you can search and find what you need. Most packages are hosted on github, but when you install a package locally into your application folder – npm figures out dependencies etc. automatically for you.

Books, glorious books

41QSvp9fTcL._SX331_BO1,204,203,200_Last but not least, get some good books. Seriously, it will save you so much time and frustration. Amazon have tons of great books, be it vanilla JavaScript, TypeScript, Node.js — pick some good ones and take the time to consume the material.

And again, I strongly urge you to have a look at Elements when it comes to WebAssembly. WebAssembly is a harsh and barren canvas, and being able to use the Elements RTL is a huge boost.

But regardless of path you pick, you will always benefit from learning vanilla JavaScript.

 

Two new groups in the Developer family

July 1, 2019 2 comments

Delphi Developer is a group on Facebook that have been going strong for 12+ years. It was one of the first groups on Facebook, created the same week that Facebook allowed groups. With that group well established, it’s time to expand and clean up the feed.

RO-Single-Gear-512Last month I introduced a new group, RemObjects Developer, which is a group for developers that use RemObjects components, like the Remoting SDK, Data Abstract and/or Hydra – but more in particular, developers using Oxygene, C#, Swift, Java or Go via Elements (RemObjects compiler toolchain).

Two new groups

To further simplify syndication, and clean up the feeds (which so far has been a pot-purrey of many topics, dialects and products) an additional two groups is now in place:

Obviously there will be some overlapping. Since FPC and Delphi has much in common and are for the most part compatible, some news will be shared between those groups. But all in all this is to clean up the newsfeed which has so far been a mix and match of everything.

org

Simple overview of the groups

Node.js Developer is not meant to be purely about vanilla JavaScript. Node.js is ultimately a JavaScript runtime-engine. Which means you can use it to run or host WebAssembly libraries (as produced by Oxygene), or generate code via DWScript or Freepascal. You can think of it as a service-host if you like.

So if you are writing WebAssembly applications using Elements, then the node.js group will no doubt be interesting too. Same goes for DWScript users, Smart Pascal users and Freepascal users – providing web tech is what they like.

What is this Quartex Components?

It’s easier to manage multiple groups if you attach them to a parent-page. So if you wonder why all the groups says “by Quartex Components”, that is just a top-level page that helps me deal with with syndication. For some reason Facebook’s API only works for pages, not groups. So it’s impossible to auto-import news (for example) without a page.

The name, “Quartex Components” is ultimately the name of my personal company. I used to produce security components for Delphi, but decided to open-source those for the community.

So Quartex Components is just an organizational element.

Porting TextCraft to Oxygene

June 30, 2019 Leave a comment

TextCraft is a simple yet powerful text parser, designed for general purpose parsing jobs. I originally implemented it for Delphi, it’s the base-parser for the LDEF bytecode assembler amongst other things. It was ported to Smart Pascal, then Freepascal – and now finally Oxygene.

ldef

The LDEF Assembler is a part of the Quartex Media Desktop

The LDEF assembler and bytecode engine is currently implemented in Smart and compiles for Javascript. It’s a complete assembler and VM allowing coders to approach Asm.js from an established instruction-set. In short: you feed it source-code, it spits out bytecodes that you can execute super fast in either the browser or elsewhere. As long as there is a VM implementation available.

The Javascript version works really well, especially on node.js. In essence, i don’t need to re-compile the toolchain when moving between arm, x86, windows, linux or osx. Think of it as a type of Java bytecodes or CLR bytecodes.

Getting the code to run under Oxygene, means that I can move the whole engine into WebAssembly. The parser, assembler and linker (et-al) can thus run as WebAssembly, and I can use that from my JavaScript front-end code. Best of both worlds – the flamboyant creativity of JavaScript, and the raw speed of WebAssembly.

The port

Before I can move over the top-level parser + assembler etc, the generic parser code has to work. I was reluctant to start because I imagined the porting would take at least a day, but luckily it took me less than an hour. There are a few superficial differences between Smart, Delphi, Freepascal and Oxygene; for example the Copy() function for strings is not a lose function in Oxygene, instead you use String.SubString(). Functions like High() and Low() on strings likewise has to be refactored.

But all in all the conversion was straight-forward, and TextCraft is now a part of the QTX library for Oxygene. I’ll be uploading a commit to GIT with the whole shabam soon.

Well, hope the WordPress parser doesnt screw this up too bad.

namespace qtxlib;

//##################################################################
// TextCraft 1.2
//  Written by Jon L. Aasenden
//
//  This is a port of TC 1.2 from Freepascal. TextCraft is initially
//  a Delphi parser framework. The original repository can be found
//  on BitBucket at:
//
//  https://bitbucket.org/hexmonks/main
//
//##################################################################

{$DEFINE USE_INCLUSIVE}
{$define USE_BMARK}

interface

uses
  qtxlib, System, rtl,
  RemObjects.Elements.RTL.Delphi,
  RemObjects.Elements.RTL.Delphi.VCL;

type

  // forward declarations
  TTextBuffer         = class;
  TParserContext      = class;
  TCustomParser       = class;
  TParserModelObject  = class;

    // Exceptions
  ETextBuffer   = class(Exception);
  EModelObject  = class(Exception);

  // Callback functions
  TTextValidCB = function (Item: Char): Boolean;

  // Bookmark datatype
  TTextBufferBookmark = class
  public
    property bbOffset: Integer;
    property bbCol:    Integer;
    property bbRow:    Integer;
    function Equals(const ThisMark: TTextBufferBookmark): Boolean;
  end;

  {.$DEFINE USE_BMARK}

  TTextBuffer = class(TErrorObject)
  private
    FData:      String;
    FOffset:    Integer;
    FLength:    Integer;
    FCol:       Integer;
    FRow:       Integer;
    {$IFDEF USE_BMARK}
    FBookmarks: List;
    {$ENDIF}
    procedure   SetCacheData(NewText: String);
  public
    property    Column: Integer read FCol;
    property    Row: Integer read FRow;
    property    Count: Integer read FLength;
    property    Offset: Integer read FOffset;
    property    CacheData: String read FData write SetCacheData;

    // These functions map directly to the "Current"
    // character where the offset is placed, and is used to
    // write code that makes more sense to human eyes
    function    CrLf: Boolean;
    function    Space: Boolean;
    function    Tab: Boolean;
    function    SemiColon: Boolean;
    function    Colon: Boolean;
    function    ConditionEnter: Boolean;
    function    ConditionLeave: Boolean;
    function    BracketEnter: Boolean;
    function    BracketLeave: Boolean;
    function    Ptr: Boolean;
    function    Punctum: Boolean;
    function    Question: Boolean;
    function    Less: Boolean;
    function    More: Boolean;
    function    Equal: Boolean;
    function    Pipe: Boolean;
    function    Numeric: Boolean;

    function    Empty: Boolean;
    function    BOF: Boolean;
    function    EOF: Boolean;
    function    Current: Char;

    function    First: Boolean;
    function    Last: Boolean;

    // Same as "Next", but does not automatically
    // consume CR+LF, used when parsing textfragments
    function    NextNoCrLf: Boolean;

    // Normal Next function, will automatically consume
    // CRLF when it encounters it
    function    Next: Boolean;

    function    Back: Boolean;

    function    Bookmark: TTextBufferBookmark;
    procedure   Restore(const Mark: TTextBufferBookmark);
    {$IFDEF USE_BMARK}
    procedure   Drop;
    {$ENDIF}

    procedure   ConsumeJunk;
    procedure   ConsumeCRLF;

    function    Compare(const CompareText: String;
                const CaseSensitive: Boolean): Boolean;

    function    Read(var Fragment: Char): Boolean; overload;
    function    Read: Char; overload;
    function    ReadTo(const CB: TTextValidCB; var TextRead: String): Boolean; overload;
    function    ReadTo(const Resignators: TSysCharSet; var TextRead: String): Boolean; overload;
    function    ReadTo(MatchText: String): Boolean; overload;
    function    ReadTo(MatchText: String; var TextRead: String): Boolean; overload;

    function    ReadToEOL: Boolean;   overload;
    function    ReadToEOL(var TextRead: String): Boolean;   overload;

    function    Peek: Char; overload;
    function    Peek(CharCount: Integer; var TextRead: String): Boolean; overload;

    function    NextNonControlChar(const CompareWith: Char): Boolean;
    function    NextNonControlText(const CompareWith: String): Boolean;

    function    ReadWord(var TextRead: String): Boolean;

    function    ReadQuotedString: String;
    function    ReadCommaList(var cList: List): Boolean;

    function    NextLine: Boolean;

    procedure   Inject(const TextToInject: String);

    function    GetCurrentLocation: TTextBufferBookmark;

    function    Trail: String;

    procedure   Clear;
    procedure   LoadBufferText(const NewBuffer: String);

    constructor Create(const BufferText: String); overload; virtual;

    finalizer;
    begin
      {$IFDEF USE_BMARK}
      FBookmarks.Clear();
      disposeAndNil(FBookmarks);
      {$endif}
      Clear();
    end;
  end;

  TParserContext = class(TErrorObject)
  private
    FBuffer:    TTextBuffer;
    FStack:     Stack;
  public
    property    Buffer: TTextBuffer read FBuffer;
    property    Model: TParserModelObject;

    procedure   Push(const ModelObj: TParserModelObject);
    function    Pop: TParserModelObject;
    function    Peek: TParserModelObject;
    procedure   ClearStack;

    constructor Create(const SourceCode: String); reintroduce; virtual;

    finalizer;
    begin
      FStack.Clear();
      FBuffer.Clear();
      disposeAndNil(FStack);
      disposeAndNil(FBuffer);
    end;
  end;

  TCustomParser = class(TErrorObject)
  private
    FContext:   TParserContext;
  protected
    procedure   SetContext(const NewContext: TParserContext);
  public
    property    Context: TParserContext read FContext;
    function    Parse: Boolean; virtual;
    constructor Create(const ParseContext: TParserContext); reintroduce; virtual;
  end;

  TParserModelObject = class(TObject)
  private
    FParent:    TParserModelObject;
    FChildren:  List;
  protected
    function    GetParent: TParserModelObject; virtual;
    function    ChildGetCount: Integer; virtual;
    function    ChildGetItem(const Index: Integer): TParserModelObject; virtual;
    function    ChildAdd(const Instance: TParserModelObject): TParserModelObject; virtual;
  public
    property    Parent: TParserModelObject read GetParent;
    property    Context: TParserContext;
    procedure   Clear; virtual;
    constructor Create(const AParent: TParserModelObject); virtual;

    finalizer;
    begin
      Clear();
      FChildren := nil;
    end;

  end;

implementation

//#####################################################################
// Error messages
//#####################################################################

const
  CNT_ERR_BUFFER_EMPTY  = 'Buffer is empty error';
  CNT_ERR_OFFSET_BOF    = 'Offset at BOF error';
  CNT_ERR_OFFSET_EOF    = 'Offset at EOF error';
  CNT_ERR_COMMENT_NOTCLOSED = 'Comment not closed error';
  CNT_ERR_OFFSET_EXPECTED_EOF = 'Expected EOF error';
  CNT_ERR_LENGTH_INVALID = 'Invalid length error';

//#####################################################################
// TTextBufferBookmark
//#####################################################################

function TTextBufferBookmark.Equals(const ThisMark: TTextBufferBookmark): boolean;
begin
  result := ( (ThisMark  nil) and (ThisMark  self) )
        and (self.bbOffset = ThisMark.bbOffset)
        and (self.bbCol = ThisMark.bbCol)
        and (self.bbRow = ThisMark.bbRow);
end;

//#####################################################################
// TTextBuffer
//#####################################################################

constructor TTextBuffer.Create(const BufferText: string);
begin
  inherited Create();
  if length(BufferText) > 0 then
    LoadBufferText(BufferText)
  else
    Clear();
end;

procedure TTextBuffer.Clear;
begin
  FData := '';
  FOffset := -1;
  FLength := 0;
  FCol := -1;
  FRow := -1;
  {$IFDEF USE_BMARK}
  FBookmarks.Clear();
  {$ENDIF}
end;

procedure TTextBuffer.SetCacheData(NewText: string);
begin
  LoadBufferText(NewText);
end;

function TTextBuffer.Trail: string;
begin
  if not Empty then
  begin
    if not EOF then
      result := FData.Substring(FOffset, length(FData) );
      //result := Copy( FData, FOffset, length(FData) );
  end;
end;

procedure TTextBuffer.LoadBufferText(const NewBuffer: string);
begin
  // Flush existing buffer
  Clear();

  // Load in buffertext, init offset and values
  var TempLen := NewBuffer.Length;
  if TempLen > 0 then
  begin
    FData := NewBuffer;
    FOffset := 0; // start at BOF
    FCol := 0;
    FRow := 0;
    FLength := TempLen;
  end;
end;

function TTextBuffer.GetCurrentLocation: TTextBufferBookmark;
begin
  if Failed then
    ClearLastError();
  if not Empty then
  begin
    result := TTextBufferBookmark.Create;
    result.bbOffset := FOffset;
    result.bbCol := FCol;
    result.bbRow := FRow;
  end else
  raise ETextBuffer.Create
  ('Failed to return position, buffer is empty error');
end;

function TTextBuffer.Bookmark: TTextBufferBookmark;
begin
  if Failed then
    ClearLastError();
  if not Empty then
  begin
    result := TTextBufferBookmark.Create;
    result.bbOffset := FOffset;
    result.bbCol := FCol;
    result.bbRow := FRow;
    {$IFDEF USE_BMARK}
    FBookmarks.add(result);
    {$ENDIF}
  end else
  raise ETextBuffer.Create
  ('Failed to bookmark location, buffer is empty error');
end;

procedure TTextBuffer.Restore(const Mark: TTextBufferBookmark);
begin
  if Failed then
    ClearLastError();
  if not Empty then
  begin
    if Mark  nil then
    begin
      FOffset := Mark.bbOffset;
      FCol := Mark.bbCol;
      FRow := Mark.bbRow;
      Mark.Free;

      {$IFDEF USE_BMARK}
      var idx := FBookmarks.Count;
      if idx > 0 then
      begin
        dec(idx);
        FOffset := FBookmarks[idx].bbOffset;
        FCol := FBookmarks[idx].bbCol;
        FRow := FBookmarks[idx].bbRow;
        FBookmarks.Remove(idx);
        //FBookmarks.SetLength(idx)
        //FBookmarks.Delete(idx,1);
      end else
      raise ETextBuffer.Create('Failed to restore bookmark, none exist');
      {$ENDIF}
    end else
    raise ETextBuffer.Create('Failed to restore bookmark, object was nil error');
  end else
  raise ETextBuffer.Create
  ('Failed to restore bookmark, buffer is empty error');
end;

{$IFDEF USE_BMARK}
procedure TTextBuffer.Drop;
begin
  if Failed then
    ClearLastError();
  if not Empty then
  begin
    if FBookmarks.Count > 0 then
      FBookmarks.Remove(FBookmarks.Count-1)
    else
      raise ETextBuffer.Create('Failed to drop bookmark, none exist');
  end else
  raise ETextBuffer.Create
  ('Failed to drop bookmark, buffer is empty error');
end;
{$ENDIF}

function TTextBuffer.Read(var Fragment: char): boolean;
begin
  if Failed then
    ClearLastError();

  if not Empty then
  begin
    result := FOffset <= length(FData);
    if result then
    begin
      // return character
      Fragment := FData[FOffset];

      // update offset
      inc(FOffset)
    end else
    begin
      // return invalid char
      Fragment := #0;

      // Set error reason
      SetLastError('Offset at BOF error');
    end;
  end else
  begin
    result := false;
    Fragment := #0;
    SetLastError('Buffer is empty error');
  end;
end;

function TTextBuffer.Read: char;
begin
  if Failed then
    ClearLastError();

  if not Empty then
  begin
    result := Current;
    Next();
  end else
  result := #0;
end;

function TTextBuffer.ReadToEOL: boolean;
begin
  if Failed then
    ClearLastError();

  if not Empty() then
  begin
    if BOF() then
    begin
      if not First() then
        exit;
    end;

    if EOF() then
    begin
      SetLastError(CNT_ERR_OFFSET_EOF);
      exit;
    end;

    // Keep start
    var LStart := FOffset;

    // Enum until match of EOF
    {$IFDEF USE_INCLUSIVE}
    repeat
      if (FData[FOffset] = #13)
      and (FData[FOffset + 1] = #10) then
      begin
        result := true;
        break;
      end else
      begin
        inc(FOffset);
        inc(FCol);
      end;
    until EOF();
    {$ELSE}
    While FOffset < High(FData) do
    begin
      if (FData[FOffset] = #13)
      and (FData[FOffset + 1] = #10) then
      begin
        result := true;
        break;
      end else
      begin
        inc(FOffset);
        inc(FCol);
      end;
    end;
    {$ENDIF}

    // Last line in textfile might not have
    // a CR+LF, so we have to check for termination
    if not result then
    begin
      if EOF then
      begin
        if LStart = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset <= high(FData) ) )
        and ( (FData[FOffset] = '= Low(FData)) and (FOffset ') );
end;

function  TTextBuffer.Equal: boolean;
begin
  result := (not Empty)
        and ( (FOffset >= Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset  LStart then
        begin
          // Any text to return? Or did we start
          // directly on a CR+LF and have no text to give?
          var LLen := FOffset - LStart;
          TextRead := FData.Substring(LStart, LLen);
          //TextRead := Copy(FData, LStart, LLen);
        end;

        // Either way, we exit because CR+LF has been found
        result := true;
        break;
      end;

      inc(FOffset);
      inc(FCol);
    until EOF();
    {$ELSE}
    While FOffset  LStart then
        begin
          // Any text to return? Or did we start
          // directly on a CR+LF and have no text to give?
          var LLen := FOffset - LStart;
          TextRead := copy(FData, LStart, LLen);
        end;

        // Either way, we exit because CR+LF has been found
        result := true;
        break;
      end;

      inc(FOffset);
      inc(FCol);
    end;
    {$ENDIF}

    // Last line in textfile might not have
    // a CR+LF, so we have to check for EOF and treat
    // that as a terminator.
    if not result then
    begin
      if FOffset >= high(FData) then
      begin
        if LStart  0 then
          begin
            TextRead := FData.Substring(LStart, LLen);
            //TextRead := Copy(FData, LStart, LLen);
            result := true;
          end;
          exit;
        end;
      end;
    end;

  end;
end;

function TTextBuffer.ReadTo(const CB: TTextValidCB; var TextRead: string): boolean;
begin
  if Failed then
    ClearLastError();

  TextRead := '';

  if not Empty then
  begin

    if BOF() then
    begin
      if not First() then
        exit;
    end;

    if EOF() then
    begin
      SetLastError(CNT_ERR_OFFSET_EOF);
      exit;
    end;

    if not assigned(CB) then
    begin
      SetLastError('Invalid callback handler');
      exit;
    end;

    {$IFDEF USE_INCLUSIVE}
    repeat
      if not CB(Current) then
        break
      else
        TextRead := TextRead + Current;

      if not Next() then
        break;
    until EOF();
    {$ELSE}
    while not EOF do
    begin
      if not CB(Current) then
        break
      else
        TextRead := TextRead + Current;

      if not Next() then
        break;
    end;
    {$ENDIF}
    result := TextRead.Length > 0;

  end else
  begin
    result := false;
    SetLastError(CNT_ERR_BUFFER_EMPTY);
  end;
end;

function TTextBuffer.ReadTo(const Resignators: TSysCharSet; var TextRead: string): boolean;
begin
  if Failed then
    ClearLastError();

  TextRead := '';
  if not Empty then
  begin

    if BOF() then
    begin
      if not First() then
        exit;
    end;

    if EOF() then
    begin
      SetLastError(CNT_ERR_OFFSET_EOF);
      exit;
    end;

    {$IFDEF USE_INCLUSIVE}
    repeat
      if not Resignators.Contains(Current) then
        TextRead := TextRead + Current
      else
        break;

      if not Next() then
        break;
    until EOF();
    {$ELSE}
    while not EOF do
    begin
      if not (Current in Resignators) then
        TextRead := TextRead + Current
      else
        break;

      if not Next() then
        break;
    end;
    {$ENDIF}

    result := TextRead.Length > 0;
  end else
  begin
    result := false;
    SetLastError(CNT_ERR_BUFFER_EMPTY);
  end;
end;

function TTextBuffer.ReadTo(MatchText: string): boolean;
begin
  if Failed then
    ClearLastError();

  if not Empty() then
  begin

    if BOF() then
    begin
      if not First() then
        exit;
    end;

    if EOF() then
    begin
      SetLastError(CNT_ERR_OFFSET_EOF);
      exit;
    end;

    var MatchLen := length(MatchText);
    if MatchLen > 0 then
    begin
      MatchText := MatchText.ToLower();

      repeat
        var TempCache := '';
        if Peek(MatchLen, TempCache) then
        begin
          TempCache := TempCache.ToLower();
          result := SameText(TempCache, MatchText);
          if result then
            break;
        end;

        if not Next then
          break;
      until EOF;
    end;

  end else
  begin
    result := false;
    SetLastError(CNT_ERR_BUFFER_EMPTY);
  end;
end;

function TTextBuffer.ReadTo(MatchText: string; var TextRead: string): boolean;
begin
  if Failed then
    ClearLastError();

  result := false;
  TextRead := '';

  if not Empty() then
  begin

    if BOF() then
    begin
      if not First() then
        exit;
    end;

    if EOF() then
    begin
      SetLastError(CNT_ERR_OFFSET_EOF);
      exit;
    end;

    if MatchText.Length > 0 then
    begin
      MatchText := MatchText.ToLower();

      repeat
        var TempCache := '';
        if Peek(MatchText.Length, TempCache) then
        begin
          TempCache := TempCache.ToLower();
          result := SameText(TempCache, MatchText);
          if result then
            break
          else
            TextRead := TextRead + Current;
        end else
          TextRead := TextRead + Current;

        if not Next() then
          break;
      until EOF;
    end;

  end else
  begin
    result := false;
    SetLastError(CNT_ERR_BUFFER_EMPTY);
  end;
end;

procedure TTextBuffer.Inject(const TextToInject: string);
begin
  if length(FData) > 0 then
  begin
    var lSeg1 := FData.Substring(1, FOffset);
    var lSeg2 := FData.Substring(FOffset + 1, length(FData));
    //var LSeg1 := Copy(FData, 1, FOffset);
    //var LSeg2 := Copy(FData, FOffset+1,  FData.Length);
    FData := lSeg1 + TextToInject + lSeg2;
  end else
    FData := TextToInject;
end;

function TTextBuffer.Compare(const CompareText: string;
    const CaseSensitive: boolean): boolean;
begin
  if Failed then
    ClearLastError();

  if not Empty() then
  begin
    if BOF() then
    begin
      if not First() then
        exit;
    end;

    if EOF() then
    begin
      SetLastError(CNT_ERR_OFFSET_EOF);
      exit;
    end;

    var LenToRead := CompareText.Length;
    if LenToRead > 0 then
    begin
      // Peek will set an error message if it
      // fails, so we dont need to set anything here
      var ReadData := '';
      if Peek(LenToRead, ReadData) then
      begin
        case CaseSensitive of
        false: result := ReadData.ToLower() = CompareText.ToLower();
        true:  result := ReadData = CompareText;
        end;
      end;
    end else
    SetLastError(CNT_ERR_LENGTH_INVALID);

  end else
  SetLastError(CNT_ERR_BUFFER_EMPTY);
end;

procedure TTextBuffer.ConsumeJunk;
begin
  if Failed then
    ClearLastError();

  if not Empty then
  begin

    if BOF() then
    begin
      if not First() then
        exit;
    end;

    if EOF() then
    begin
      SetLastError(CNT_ERR_OFFSET_EOF);
      exit;
    end;

    repeat
      case Current of
      ' ':
        begin
        end;
      '"':
        begin
          break;
        end;
      #8, #09:
        begin
        end;
      '/':
        begin
          (* Skip C style remark *)
          if Compare('/*', false) then
          begin
            if ReadTo('*/') then
            begin
              inc(FOffset, 2);
              Continue;
            end else
            SetLastError(CNT_ERR_COMMENT_NOTCLOSED);
          end else
          begin
            (* Skip Pascal style remark *)
            if Compare('//', false) then
            begin
              if ReadToEOL() then
              begin
                continue;
              end else
              SetLastError(CNT_ERR_OFFSET_EXPECTED_EOF);
            end;
          end;
        end;
      '(':
        begin
          (* Skip pascal style remark *)
          if Compare('(*', false)
            and not Compare('(*)', false) then
          begin
            if ReadTo('*)') then
            begin
              inc(FOffset, 2);
              continue;
            end else
            SetLastError(CNT_ERR_COMMENT_NOTCLOSED);
          end else
          break;
        end;
      #13:
        begin
          if FData[FOffset + 1] = #10 then
            inc(FOffset, 2)
          else
            inc(FOffset, 1);
          //if Peek = #10 then
          //  ConsumeCRLF;
          continue;
        end;
      #10:
        begin
          inc(FOffset);
          continue;
        end;
      else
        break;
      end;

      if not Next() then
        break;
    until EOF;

  end else
  SetLastError(CNT_ERR_BUFFER_EMPTY);
end;

procedure TTextBuffer.ConsumeCRLF;
begin
  if not Empty then
  begin

    if BOF() then
    begin
      if not First() then
        exit;
    end;

    if EOF() then
    begin
      SetLastError(CNT_ERR_OFFSET_EOF);
      exit;
    end;

    if  (FData[FOffset] = #13) then
    begin
      if FData[FOffset + 1] = #10 then
        inc(FOffset, 2)
      else
        inc(FOffset);

      inc(FRow);
      FCol := 0;
    end;

  end;
end;

function TTextBuffer.Empty: boolean;
begin
  result := FLength < 1;
end;

// This method will look ahead, skipping space, tab and crlf (also known
// as control characters), and when a non control character is found it will
// perform a string compare. This method uses a bookmark and will restore
// the offset to the same position as when it was entered.
//
// Notes: The method "NextNonControlChar" is a similar method that
// performs a char-only compare.
function TTextBuffer.NextNonControlText(const CompareWith: string): boolean;
begin
  if Failed then
    ClearLastError();

  if not Empty then
  begin

    if BOF() then
    begin
      if not First() then
        exit;
    end;

    if EOF() then
    begin
      SetLastError(CNT_ERR_OFFSET_EOF);
      exit;
    end;

    var Mark := Bookmark();
    try
      // Iterate ahead
      repeat
        if not (Current in [' ', #13, #10, #09]) then
          break;

        Next();
      until EOF();

      // Compare unless we hit the end of the line
      if not EOF then
        result := Compare(CompareWith, false);
    finally
      Restore(Mark);
    end;

  end else
  SetLastError(CNT_ERR_BUFFER_EMPTY);
end;

// This method will look ahead, skipping space, tab and crlf (also known
// as control characters), and when a non control character is found it will
// perform a string compare. This method uses a bookmark and will restore
// the offset to the same position as when it was entered.

function TTextBuffer.NextNonControlChar(const CompareWith: char): boolean;
begin
  if Failed then
    ClearLastError();

  if not Empty then
  begin
    if BOF() then
    begin
      if not First() then
        exit;
    end;

    if EOF() then
    begin
      SetLastError(CNT_ERR_OFFSET_EOF);
      exit;
    end;

    var Mark := Bookmark();
    try
      repeat
        if not (Current in [' ', #13, #10, #09]) then
          break;
        Next();
      until EOF();

      //if not EOF then
      result := Current.ToLower() = CompareWith.ToLower();
      //result := LowerCase(Current) = LowerCase(CompareWith);

    finally
      Restore(Mark);
    end;

  end else
  SetLastError(CNT_ERR_BUFFER_EMPTY);
end;

function TTextBuffer.Peek: char;
begin
  if Failed then
    ClearLastError();
  if not Empty then
  begin
    if (FOffset  0 do
        begin
          TextRead := TextRead + Current;
          if not Next() then
            break;
          dec(CharCount);
        end;
      finally
        Restore(Mark);
      end;

      result := TextRead.Length > 0;

    end else
    SetLastError(CNT_ERR_OFFSET_EOF);
  end else
  SetLastError(CNT_ERR_BUFFER_EMPTY);
end;

function TTextBuffer.First: boolean;
begin
  if Failed then
    ClearLastError();

  if not Empty then
  begin
    FOffset := Low(FData);
    result := true;
  end else
  SetLastError(CNT_ERR_BUFFER_EMPTY);
end;

function TTextBuffer.Last: boolean;
begin
  if Failed then
    ClearLastError();

  if not Empty then
  begin
    FOffset := high(FData);
    result := true;
  end else
  SetLastError(CNT_ERR_BUFFER_EMPTY);
end;

function TTextBuffer.NextNoCrLf: boolean;
begin
  if Failed then
    ClearLastError();

  if not Empty then
  begin
    // Check that we are not EOF
    result := FOffset <= high(FData);
    if result then
    begin
      // Update offset into buffer
      inc(FOffset);

      // update column, but not if its in a lineshift
      if not (FData[FOffset] in [#13, #10]) then
        inc(FCol);

    end else
    SetLastError(CNT_ERR_OFFSET_EOF);
  end else
  SetLastError(CNT_ERR_BUFFER_EMPTY);
end;

function TTextBuffer.Next: boolean;
begin
  if Failed then
    ClearLastError();

  if not Empty() then
  begin

    if BOF() then
    begin
      if not First() then
        exit;
    end;

    if EOF() then
    begin
      SetLastError(CNT_ERR_OFFSET_EOF);
      exit;
    end;

    // Update offset into buffer
    inc(FOffset);

    // update column
    inc(FCol);

    // This is the same as ConsumeCRLF
    // But this does not generate any errors since we PEEK
    // ahead into the buffer to make sure the combination
    // is correct before we adjust the ROW + offset
    if FOffset  Low(FData));
    if result then
      dec(FOffset)
    else
      SetLastError(CNT_ERR_OFFSET_BOF);
  end else
  SetLastError(CNT_ERR_BUFFER_EMPTY);
end;

function TTextBuffer.Current: char;
begin
  if Failed then
    ClearLastError();

  // Check that buffer is not empty
  if not Empty then
  begin
    // Check that we are on char 1 or more
    if FOffset >= Low(FData) then
    begin
      // Check that we are before or on the last char
      if (FOffset <= high(FData)) then
        result := FData[FOffset]
      else
      begin
        SetLastError(CNT_ERR_OFFSET_EOF);
        result := #0;
      end;
    end else
    begin
      SetLastError(CNT_ERR_OFFSET_BOF);
      result := #0;
    end;
  end else
  begin
    SetLastError(CNT_ERR_BUFFER_EMPTY);
    result := #0;
  end;
end;

function TTextBuffer.BOF: boolean;
begin
  if not Empty then
    result := FOffset  high(FData);
end;

function TTextBuffer.NextLine: boolean;
begin
  if Failed then
    ClearLastError();

  if not Empty then
  begin
    // Make sure we offset to a valid character
    // in the buffer.
    ConsumeJunk();

    if not EOF then
    begin
      var ThisRow := self.FRow;
      while Row = ThisRow do
      begin
        Next();
        if EOF then
        break;
      end;

      result := (Row  ThisRow) and (not EOF);
    end;
  end;
end;

function TTextBuffer.ReadWord(var TextRead: string): boolean;
begin
  if Failed then
    ClearLastError();

  TextRead := '';

  if not Empty then
  begin
    // Make sure we offset to a valid character
    // in the buffer.
    ConsumeJunk();

    // Not at the end of the file?
    if not EOF then
    begin
      repeat
        var el := Current;

        if (el in
        [ 'A'..'Z',
          'a'..'z',
          '0'..'9',
          '_', '-' ]) then
          TextRead := TextRead + el
        else
          break;

        if not NextNoCrLf() then
          break;

      until EOF;

      result := TextRead.Length > 0;

    end else
    SetLastError('Failed to read word, unexpected EOF');
  end else
  SetLastError('Failed to read word, buffer is empty error');
end;

function TTextBuffer.ReadCommaList(var cList: List): boolean;
var
  LTemp: String;
  LValue: String;
begin
  if cList = nil then
    cList := new List
  else
    cList.Clear();

  if not Empty then
  begin
    ConsumeJunk();

    While not EOF do
    begin
      case Current of
      #09:
        begin
          // tab, just skip
        end;
      #13, #10:
        begin
          // CR+LF, consume and continue;
          ConsumeCRLF();
        end;
      #0:
        begin
          // Unexpected EOL
          break;
        end;

      ';':
        begin
          //Perfectly sound ending
          result := true;
          break;
        end;
      '"':
        begin
          LValue := ReadQuotedString;
          if LValue.Length > 0 then
          begin
            cList.add(LValue);
            LValue := '';
          end;
        end;
      ',':
        begin
          LTemp := LTemp.Trim();
          if LTemp.Length>0 then
          begin
            cList.add(LTemp);
            LTemp := '';
          end;
        end;
      else
        begin
          LTemp := LTemp + Current;
        end;
      end;

      if not Next() then
        break;
    end;

    if LTemp.Length > 0 then
      cList.add(LTemp);

    result := cList.Count > 0;

  end;
end;

function TTextBuffer.ReadQuotedString: string;
begin
  if not Empty then
  begin
    if not EOF then
    begin

      // Make sure we are on the " entry quote
      if Current  '"' then
      begin
        SetLastError('Failed to read quoted string, expected index on " character error');
        exit;
      end;

      // Skip the entry char
      if not NextNoCrLf() then
      begin
        SetLastError('Failed to skip initial " character error');
        exit;
      end;

      while not EOF do
      begin
        // Read char from buffer
        var TempChar := Current;

        // Closing of string? Exit
        if TempChar = '"' then
        begin
          if not NextNoCrLf then
            SetLastError('failed to skip final " character in string error');
          break;
        end;

        result := result + TempChar;

        if not NextNoCrLf() then
          break;
      end;

    end;
  end;
end;

//##########################################################################
// TParserModelObject
//##########################################################################

constructor TParserModelObject.Create(const AParent:TParserModelObject);
begin
  inherited Create;
  FParent := AParent;
  FChildren := new List;
end;

function TParserModelObject.GetParent:TParserModelObject;
begin
  result := FParent;
end;

procedure TParserModelObject.Clear;
begin
  FChildren.Clear();
end;

function TParserModelObject.ChildGetCount: integer;
begin
  result := FChildren.Count;
end;

function TParserModelObject.ChildGetItem(const Index: integer): TParserModelObject;
begin
  result := TParserModelObject(FChildren[Index]);
end;

function TParserModelObject.ChildAdd(const Instance: TParserModelObject): TParserModelObject;
begin
  if FChildren.IndexOf(Instance) < 0 then
    FChildren.add(Instance);
  result := Instance;
end;

//###########################################################################
// TParserContext
//###########################################################################

constructor TParserContext.Create(const SourceCode: string);
begin
  inherited Create;
  FBuffer := TTextBuffer.Create(SourceCode);
  FStack := new Stack;
end;

procedure TParserContext.Push(const ModelObj: TParserModelObject);
begin
  if Failed then
    ClearLastError();

  try
    FStack.Push(ModelObj);
  except
    on e: Exception do
    SetLastError('Internal error:' + e.Message);
  end;
end;

function TParserContext.Pop: TParserModelObject;
begin
  if Failed then
    ClearLastError();
  try
    result := FStack.Pop();
  except
    on e: Exception do
    SetLastError('Internal error:' + e.Message);
  end;
end;

function TParserContext.Peek: TParserModelObject;
begin
  if Failed then
    ClearLastError();
  try
    result := FStack.Peek();
  except
    on e: Exception do
    SetLastError('Internal error:' + e.Message);
  end;
end;

procedure TParserContext.ClearStack;
begin
  if Failed then
    ClearLastError();
  try
    FStack.Clear();
  except
    on e: Exception do
    SetLastError('Internal error:' + e.Message);
  end;
end;

//###########################################################################
// TCustomParser
//###########################################################################

constructor TCustomParser.Create(const ParseContext: TParserContext);
begin
  inherited Create;
  FContext := ParseContext;
end;

function TCustomParser.Parse: boolean;
begin
  result := false;
  SetLastErrorF('No parser implemented for class %s',[ClassName]);
end;

procedure TCustomParser.SetContext(const NewContext: TParserContext);
begin
  FContext := NewContext;
end;

end.

Generic protect for FPC/Lazarus

June 30, 2019 Leave a comment

Freepascal is not frequently mentioned on my blog. I have written about it from time to time, not always in a positive light though. Just to be clear, FPC (the compiler) is fantastic; it was one particular fork of Lazarus I had issues with, involving a license violation.

On the whole, freepascal and Lazarus is capable of great things. There are a few quirks here and there (if not oddities) that prevents mass adoption (the excessive use of include-files to “fake” partial classes being one), but as object-pascal compilers go, Freepascal is a battle-hardened, production ready system.

It’s been Linux in particular that I have used Freepascal on. In 2015 Hydro Oil wanted to move their back-end from Windows to Linux, and I spent a few months converting windows-only services into Linux daemons.

Today I find myself converting parts of the toolkit I came up with to Oxygene, but that’s a post for another day.

Generic protect

If you work a lot with multithreaded code, the unit im posting here might come in handy. Long story short: sharing composite objects between threads and the main process, always means extra scaffolding. You have to make sure you don’t access the list (or it’s elements) at the same time as another thread for example. To ensure this you can either use a critical-section, or you can deliver the data with a synchronized call. This is more or less universal for all languages, no matter if you are using Oxygene, C/C++, C# or Delphi.

When this unit came into being, I was doing quite elaborate classes with a lot of lists. These classes could not share ancestor, or I could have gotten away with just one locking mechanism. Instead I had to implement the same boilerplate code over and over again.

The unit below makes insulating (or protecting) classes easier. It essentially envelopes whatever class-instance you feed it, and returns the proxy object. Whenever you want to access your instance, you have to unlock it first or use a synchronizer (see below).

Works in both Freepascal and Delphi

The unit works for both Delphi and Freepascal, but there is one little difference. For some reason Freepascal does not support anonymous procedures, so we compensate and use inline-procedures instead. While not a huge deal, I really hope the FPC team add anonymous procedures, it makes life a lot easier for generics based code. Async programming without anonymous procedures is highly impractical too.

So if you are in Delphi you can write:

var
 lValue: TProtectedValue;
 lValue.Synchronize( procedure (var Value: integer)
 begin
   Value := Value * 12;
 end);

But under Freepascal you must resort to:

var
 lValue: TProtectedValue;

procedure _UpdateValue(var Data: integer);
begin
 Data := Data * 12;
end;

begin
  lValue.Synchronize(@_UpdateValue);
end;

On small examples like these, the benefit of this style of coding might be lost; but if you suddenly have 40-50 lists that needs to be shared between 100-200 active threads, it will be a time saver!

You can also use it on intrinsic datatypes:

lazarus

OK, here we go:

unit safeobjects;

// 	SafeObjects
//	==========================================================================
//	Written by Jon-Lennart Aasenden
//	Copyright Quartex Components LTD, all rights reserved
//
//	This unit is a part of the QTX Patreon Library
//
//	NOTES ABOUT FREEPASCAL:
//	=======================
//	Freepascal does not allow anonymous procedures, which means we must
//	resort to inline procedures instead:
//
// 	Where we in Delphi could write the following for an atomic,
//	thread safe alteration:
//
// var
// 	LValue: TProtectedValue;
//
//	LValue.Synchronize( procedure (var Value: integer)
//	begin
//		Value := Value * 12;
//	end);
//
//	Freepascal demands that we use an inline procedure instead, which
//  is more or less the same code, just organized slightly differently.
//
// var
// 	LValue: TProtectedValue;
//
//  procedure _UpdateValue(var Data: integer);
//  begin
//  	Data := Data * 12;
//  end;
//
// begin
//	LValue.Synchronize(@_UpdateValue);
// end;
//
//
//
//

{$mode DELPHI}
{$H+}

interface

uses
  {$IFDEF FPC}
  SysUtils,
  Classes,
  SyncObjs,
  Generics.Collections;
	{$ELSE}
  System.SysUtils,
  System.Classes,
  System.SyncObjs,
  System.Generics.Collections;
  {$ENDIF}

type

  {$DEFINE INHERIT_FROM_CRITICALSECTION}

  TProtectedValueAccessRights = set of (lvRead, lvWrite);

  EProtectedValue = class(exception);
  EProtectedObject = class(exception);

  (* Thread safe intrinsic datatype container.
     When sharing values between processes, use this class
     to make read/write access safe and protected. *)

  {$IFDEF INHERIT_FROM_CRITICALSECTION}
  TProtectedValue = class(TCriticalSection)
  {$ELSE}
  TProtectedValue = class(TObject)
  {$ENDIF}
  strict private
    {$IFNDEF INHERIT_FROM_CRITICALSECTION}
    FLock: TCriticalSection;
    {$ENDIF}
    FData: T;
    FOptions: TProtectedValueAccessRights;
  strict protected
    function GetValue: T;virtual;
    procedure SetValue(Value: T);virtual;
    function GetAccessRights: TProtectedValueAccessRights;
    procedure SetAccessRights(Rights: TProtectedValueAccessRights);
  public
    type
  		{$IFDEF FPC}
      TProtectedValueEntry = procedure (var Data: T);
  		{$ELSE}
      TProtectedValueEntry = reference to procedure (var Data: T);
      {$ENDIF}
  public
    constructor Create(Value: T); overload; virtual;
    constructor Create(Value: T; const Access: TProtectedValueAccessRights); overload; virtual;
    constructor Create(const Access: TProtectedValueAccessRights); overload; virtual;
    destructor Destroy;override;

    {$IFNDEF INHERIT_FROM_CRITICALSECTION}
    procedure Enter;
    procedure Leave;
    {$ENDIF}
    procedure Synchronize(const Entry: TProtectedValueEntry);

    property AccessRights: TProtectedValueAccessRights read GetAccessRights;
    property Value: T read GetValue write SetValue;
  end;

  (* Thread safe object container.
     NOTE #1: This object container **CREATES** the instance and maintains it!
              Use Edit() to execute a protected block of code with access
              to the object.

     Note #2: SetValue() does not overwrite the object reference, but
              attempts to perform TPersistent.Assign(). If the instance
              does not inherit from TPersistent an exception is thrown. *)
  TProtectedObject = class(TObject)
  strict private
    FData:      T;
    FLock:      TCriticalSection;
    FOptions:   TProtectedValueAccessRights;
  strict protected
    function    GetValue: T;virtual;
    procedure   SetValue(Value: T);virtual;
    function    GetAccessRights: TProtectedValueAccessRights;
    procedure   SetAccessRights(Rights: TProtectedValueAccessRights);
  public
    type
			{$IFDEF FPC}
      TProtectedObjectEntry = procedure (const Data: T);
	    {$ELSE}
      TProtectedObjectEntry = reference to procedure (const Data: T);
      {$ENDIF}
  public
    property    Value: T read GetValue write SetValue;
    property    AccessRights: TProtectedValueAccessRights read GetAccessRights;

    function    Lock: T;
    procedure   Unlock;
    procedure   Synchronize(const Entry: TProtectedObjectEntry);

    Constructor Create(const AOptions: TProtectedValueAccessRights = [lvRead,lvWrite]); virtual;
    Destructor  Destroy; override;
  end;

  (* TProtectedObjectList:
     This is a thread-safe object list implementation.
     It works more or less like TThreadList, except it deals with objects *)
  TProtectedObjectList = class(TInterfacedPersistent)
  strict private
    FObjects: TObjectList;
    FLock: TCriticalSection;
  strict protected
    function GetEmpty: boolean;virtual;
    function GetCount: integer;virtual;

    (* QueryObject Proxy: TInterfacedPersistent allows us to
       act as a proxy for QueryInterface/GetInterface. Override
       and provide another child instance here to expose
       interfaces from that instread *)
  protected
    function GetOwner: TPersistent;override;

  public
    type
      {$IFDEF FPC}
      TProtectedObjectListProc = procedure (Item: TObject; var Cancel: boolean);
      {$ELSE}
      TProtectedObjectListProc = reference to procedure (Item: TObject; var Cancel: boolean);
      {$ENDIF}
  public
    constructor Create(OwnsObjects: Boolean = true); virtual;
    destructor  Destroy; override;

    function    Contains(Instance: TObject): boolean; virtual;
    function    Enter: TObjectList; virtual;
    Procedure   Leave; virtual;
    Procedure   Clear; virtual;

    procedure   ForEach(const CB: TProtectedObjectListProc); virtual;

    Property    Count: integer read GetCount;
    Property    Empty: boolean read GetEmpty;
  end;

implementation

//############################################################################
//  TProtectedObjectList
//############################################################################

constructor TProtectedObjectList.Create(OwnsObjects: Boolean = True);
begin
  inherited Create;
  FObjects := TObjectList.Create(OwnsObjects);
  FLock := TCriticalSection.Create;
end;

destructor TProtectedObjectList.Destroy;
begin
  FLock.Enter;
  FObjects.Free;
  FLock.Free;
  inherited;
end;

procedure TProtectedObjectList.Clear;
begin
  FLock.Enter;
  try
    FObjects.Clear;
  finally
    FLock.Leave;
  end;
end;

function TProtectedObjectList.GetOwner: TPersistent;
begin
  result := NIL;
end;

procedure TProtectedObjectList.ForEach(const CB: TProtectedObjectListProc);
var
  LItem:  TObject;
  LCancel:  Boolean;
begin
	LCancel := false;
  if assigned(CB) then
  begin
    FLock.Enter;
    try
    	{$HINTS OFF}
      for LItem in FObjects do
      begin
        LCancel := false;
        CB(LItem, LCancel);
        if LCancel then
        	break;
      end;
      {$HINTS ON}
    finally
      FLock.Leave;
    end;
  end;
end;

function TProtectedObjectList.Contains(Instance: TObject): boolean;
begin
  result := false;
  if assigned(Instance) then
  begin
    FLock.Enter;
    try
      result := FObjects.Contains(Instance);
    finally
      FLock.Leave;
    end;
  end;
end;

function TProtectedObjectList.GetCount: integer;
begin
  FLock.Enter;
  try
    result :=FObjects.Count;
  finally
    FLock.Leave;
  end;
end;

function TProtectedObjectList.GetEmpty: Boolean;
begin
  FLock.Enter;
  try
    result := FObjects.Count<1;
  finally
    FLock.Leave;
  end;
end;

function TProtectedObjectList.Enter: TObjectList;
begin
  FLock.Enter;
  result := FObjects;
end;

procedure TProtectedObjectList.Leave;
begin
  FLock.Leave;
end;

//############################################################################
//  TProtectedObject
//############################################################################

constructor TProtectedObject.Create(const AOptions: TProtectedValueAccessRights = [lvRead, lvWrite]);
begin
  inherited Create;
  FLock := TCriticalSection.Create;
  FLock.Enter();
  try
  	FOptions := AOptions;
  	FData := T.Create;
  finally
    FLock.Leave();
  end;
end;

destructor TProtectedObject.Destroy;
begin
	FData.free;
  FLock.Free;
  inherited;
end;

function TProtectedObject.GetAccessRights: TProtectedValueAccessRights;
begin
  FLock.Enter;
  try
    result := FOptions;
  finally
    FLock.Leave;
  end;
end;

procedure TProtectedObject.SetAccessRights(Rights: TProtectedValueAccessRights);
begin
  FLock.Enter;
  try
    FOptions := Rights;
  finally
    FLock.Leave;
  end;
end;

function TProtectedObject.Lock: T;
begin
  FLock.Enter;
  result := FData;
end;

procedure TProtectedObject.Unlock;
begin
  FLock.Leave;
end;

procedure TProtectedObject.Synchronize(const Entry: TProtectedObjectEntry);
begin
  if assigned(Entry) then
  begin
    FLock.Enter;
    try
      Entry(FData);
    finally
      FLock.Leave;
    end;
  end;
end;

function TProtectedObject.GetValue: T;
begin
  FLock.Enter;
  try
    if (lvRead in FOptions) then
    	result := FData
  	else
    	raise EProtectedObject.CreateFmt('%s:Read not allowed error',[classname]);
  finally
    FLock.Leave;
  end;
end;

procedure TProtectedObject.SetValue(Value: T);
begin
  FLock.Enter;
  try
    if (lvWrite in FOptions) then
    begin
      if (TObject(FData) is TPersistent)
      or (TObject(FData).InheritsFrom(TPersistent)) then
      	TPersistent(FData).Assign(TPersistent(Value))
    	else
      	raise EProtectedObject.CreateFmt
        	('Locked object assign failed, %s does not inherit from %s',
        	[TObject(FData).ClassName,'TPersistent']);

    end else
    raise EProtectedObject.CreateFmt('%s:Write not allowed error',[classname]);
  finally
    FLock.Leave;
  end;
end;

//############################################################################
//  TProtectedValue
//############################################################################

Constructor TProtectedValue.Create(const Access: TProtectedValueAccessRights);
begin
  inherited Create;
  {$IFNDEF INHERIT_FROM_CRITICALSECTION}
  FLock := TCriticalSection.Create;
  {$ENDIF}
  FOptions := Access;
end;

constructor TProtectedValue.Create(Value: T);
begin
  inherited Create;
  {$IFNDEF INHERIT_FROM_CRITICALSECTION}
  FLock := TCriticalSection.Create;
  {$ENDIF}
  FOptions := [lvRead, lvWrite];
  FData := Value;
end;

constructor TProtectedValue.Create(Value: T; const Access: TProtectedValueAccessRights);
begin
  inherited Create;
  {$IFNDEF INHERIT_FROM_CRITICALSECTION}
  FLock := TCriticalSection.Create;
  {$ENDIF}
  FOptions := Access;
  FData := Value;
end;

Destructor TProtectedValue.Destroy;
begin
  {$IFNDEF INHERIT_FROM_CRITICALSECTION}
  FLock.Free;
  {$ENDIF}
  inherited;
end;

function TProtectedValue.GetAccessRights: TProtectedValueAccessRights;
begin
  Enter();
  try
    result := FOptions;
  finally
    Leave();
  end;
end;

procedure TProtectedValue.SetAccessRights(Rights: TProtectedValueAccessRights);
begin
  Enter();
  try
    FOptions := Rights;
  finally
    Leave();
  end;
end;

{$IFNDEF INHERIT_FROM_CRITICALSECTION}
procedure TProtectedValue.Enter;
begin
  FLock.Enter;
end;

procedure TProtectedValue.Leave;
begin
  FLock.Leave;
end;
{$ENDIF}

procedure TProtectedValue.Synchronize(const Entry: TProtectedValueEntry);
begin
  if assigned(Entry) then
  Begin
    Enter();
    try
      Entry(FData);
    finally
      Leave();
    end;
  end;
end;

function TProtectedValue.GetValue: T;
begin
  Enter();
  try
    if (lvRead in FOptions) then
    	result := FData
    else
    	raise EProtectedValue.CreateFmt('%s: Read not allowed error', [Classname]);
  finally
    Leave();
  end;
end;

procedure TProtectedValue.SetValue(Value: T);
begin
  Enter();
  try
    if (lvWrite in FOptions) then
    	FData:=Value
    else
    	raise EProtectedValue.CreateFmt('%s: Write not allowed error', [Classname]);
  finally
    Leave();
  end;
end;

end.

Raspberry PI 4 at last!

June 25, 2019 2 comments

It was with astonishment that I opened up my browser this morning to read some daily IT news, only to discover that the Raspberry PI v4 has finally arrived! And boy what a landslide update to the 3.x family it is!

Three times the fun

There are plenty of sites that entertains page-up and page-down with numbers, but I will save all that for an article where I have the physical kit in my posession. But looking at the preliminaries I think it’s safe to say that we are looking at a solid 3x the speed of the older yet capable PI 3b+.

Raspberry-Pi-4-4

The PI returns, and what a joy it is!

While the 3x speed boost is enough to bump the SoC up, from entertaining to serious for business applications – it’s ultimately the memory footprint that will make all the difference. While the Raspberry PI is probably the most loved SBC (single board computer) of all time, it’s always been cut short due to lack of memory. 512 megabyte can only do so much in 2019, and even the slimmest of Linux distributions quickly consumes more ram that older versions could supply.

VideoCore 6, two screens and 4k video

The new model ships in three different configurations, with 1, 2 and 4 gigabytes of ram respectively. I strongly urge people to get the 4Gb version, because with that amount of memory coupled with a good solid-state-disk, means you can enable a proper swap-partition. No matter how fast a SoC might be, without memory to compliment it – the system simply wont be able to deliver on its potential. But with 4Gb, a nice solid state disk (just use a SSD-To-USB with one of the sexy new USB 3.x ports) and you are looking at an OK mini-computer capable of most desktop applications.

I have to admit I never expected the PI to ship with support for two monitors, but lo-and-behold, the board has two mini-hdmi out ports! The board is also fitted with the VideCore 6 rather than VideoCore 4.

Not missing the boat with Oxygene and Elements

One of the most frustrating episodes in the history of Delphi, is that we never got a Delphi edition that could target Raspberry PI (or ARM-Linux in general). It was especially frustrating since Allen Bauer actually demonstrated Delphi generating code that ran on a PI in 2012. The result of not directly supporting the PI, even on service level without a UI layer – is that Delphi developers have missed the IOT market completely.

Before Delphi developers missed the IOT revolution, Delphi also missed out on iOS and Android. By the time Delphi developers could target any of these platforms, the market was completely saturated, and all opportunities to make money was long gone. In other words, Delphi has missed the boat on 3 revolutionary platforms in a row. Something which is borderline unforgivable.

The good news though is that Oxygene, the object-pascal compiler from RemObjects, supports the Raspberry PI SoC. I have yet to test this on v4, but since the PI v4 is 100% backwards compatible I don’t see any reason why there should be any issues. The code generated by Oxygene is not bound to just the PI either. As long as it runs on a debian based distro, it should run just fine on most ARM-Linux SoC’s that have working drivers.

And like I have written about elsewhere, you can also compile for WebAssembly, running either in node.js or in the browser — so there are plenty of ways to get your products over!

Stay tuned for the lineup

This week im going to do a lot of testing on various ARM devices to find out just how many SBC’s Oxygene can target, starting with the ODroid N2. But for Raspberry PI, that should be a slam-dunk. Meaning that object-pascal developers can finally make use of affordable off-the-shelves parts in their hardware projects.

As of writing im preparing the various boards I will be testing. We have the PI 3b+, the Tinkerboard from ASUS, NanoPI, Dragonboard, Odroid XU4 – and the latest power-board, the ODroid N2. Out of these offerings only the N2 is en-par with the Raspberry PI v4, although I suspect the Videocore 6 GPU will outperform the Mali G52.

Hydra now supports Freepascal and Java

June 15, 2019 2 comments

In case you guys missed it, RemObjects Hydra 6.2 now supports FreePascal!

This means that you can now use forms and units from .net and Java from your Freepascal applications – and (drumroll) also mix and match between Delphi, .net, Java and FPC modules! So if you see something cool that Freepascal lacks, just slap it in a Hydra module and you can use it across language barriers.

I have used Hydra for years with Delphi, and being able to use .net forms and components in Delphi is pretty awesome. It’s also a great framework for building modular applications that are easier to manage.

64296152_10156282425590906_705072396930908160_n

Being able to tap into Freepascal is a great feature. Or the other way around, with Freepascal showing forms from Delphi, .net or Java.

For example, if you are moving to Freepascal, you can isolate the forms or controls that are not available under Freepascal in a Hydra module, and voila – you can gradually migrate.

If you are moving to Oxygene Pascal the same applies, you can implement the immediate logic under .net, and then import and use the parts that can’t easily be ported (or that you want to wait with).

The best of four worlds — You gotta love that!

Check out Hydra here:

https://hydra.remobjects.com/hydra/whatsnew/default.aspx

 

RemObjects VCL, mind blown!

June 12, 2019 12 comments

For a guy that spends most of his time online, and can talk for hours about the most nerdy topics known to mankind – being gobsmacked and silenced is a rare event. But this morning that was exactly what happened.

Now, Marc Hoffman has blogged regularly over the years regarding the evolution of the RemObjects toolchain; explaining how they decoupled the parts that make up a programming language, such as syntax, rtl and target, but I must admit haven’t really digested the full implications of that work.

Like most developers I have kept my eyes on the parts relevant for me, like the Remoting SDK, Data Abstract and Javascript support. Before I worked at Embarcadero I pretty much spent 10 years contracting -and building Smart Mobile Studio on the side together with the team at The Smart Company Inc.

xo

Smart Pascal gained support for RemObjects SDK servers quite early

Since both the Remoting SDK and Data Abstract were part of our toolbox as Delphi developers, those were naturally more immediate than anything else. We also added support for RemObjects Remoting SDK inside Smart Mobile Studio, so that people could call existing services from their Javascript applications.

Oxygene then

Like most Delphi developers I remember testing Oxygene Pascal when I bought Delphi 2005. Back then Oxygene was licensed by Borland under the “Prism” name and represented their take on dot net support. I was very excited when it came out, but since my knowledge of the dot net framework was nil, I was 100% relient on the documentation.

In many ways Oxygene was a victim of Rad Studio’s abhorrent help-file system. Documentation for Rad Studio (especially Delphi) up to that point had been exemplary since Delphi 4; but by the time Rad Studio 2005 came out, the bloat had reached epic levels. Even for me as a die-hard Delphi fanatic, Delphi 2005 and 2006 was a tragic experience.

image

Removing Oxygene was a monumental mistake

I mean, when it takes 15 minutes (literally) just to open the docs, then learning a whole new programming paradigm under those conditions was quite frankly impossible. Like most Delphi developers I was used to Delphi 7 style documentation, where the docs were not just reference material – but actually teaches you the language itself.

In the end Oxygene remained very interesting, but with a full time job, deadlines and kids to take care of, I stuck to what I knew – namely the VCL.

Oxygene today

Just like Delphi has evolved and improved radically since 2005, Oxygene has likewise evolved above and beyond its initial form. Truth be told, we copied a lot of material from Oxygene when we made Smart Pascal, so I feel strangely at home with Oxygene even after a couple of days. The documentation for Oxygene Pascal (and Elements as a whole) is very good: https://docs.elementscompiler.com/Oxygene/

But Oxygene Pascal, while the obvious “first stop” for Delphi developers looking to expand their market impact, is more than “just a language”. It’s a language that is a part of a growing family of languages that RemObjects support and evolve.

As of writing RemObjects offers the following languages. So even if you don’t have a background in Delphi, or perhaps migrated from Delphi to C# years ago – RemObjects will have solutions and benefits to offer:

  • Oxygene (object pascal)
  • C#
  • Swift
  • Java
water

Water is a sexy, slim new IDE for RemObjects languages on Windows. For the OS X version you want to download Fire.

And here is the cool thing: when you hear “Java” you automatically expect that you are bound hands and feet to the Java runtime-libraries right? Same also with C#, you expect C# to be purely limited to the dot-net framework. And if you like me dabbed in Oxygene back in 2005-2006, you probably think Oxygene is purely a dot-net adapted version of Object Pascal right? But RemObjects have turned that on it’s head!

Remember the decoupling I mentioned at the beginning of this post? What that means in practical terms is that they have separated each language into three distinct parts:

  1. The syntax
  2. The RTL
  3. The target

What this means, is that you can pick your own combinations!

Let’s say you are coming from Delphi. You have 20 years of Object Pascal experience under your belt, and while you dont mind learning new things – Object Pascal is where you will be most productive.

Well in that case picking Oxygene Pascal covers the syntax part. But you don’t have to use the dot-net framework if you don’t want to. You can mix and match these 3 parts as you see fit! Let’s look at some combinations you could pick:

  • Oxygene Pascal -> dot net framework -> CIL
  • Oxygene Pascal -> “VCL” -> CIL
  • Oxygene Pascal -> “VCL” -> WinAPI
  • Oxygene Pascal -> “VCL” -> WebAssembly

(*) The “VCL” here is a compatibility RTL closely modeled on the Freepascal LCL and Delphi VCL. This is written from scratch and contains no proprietary code. It is purely to get people productive faster.

The whole point of this tripartite decoupling is to allow developers to maximize the value of their existing skill-set. If you know Object Pascal then that is a natural starting point for you. If you know the VCL then obviously the VCL compatibility RTL is going to help you become productive much faster than calling WinAPI on C level. But you can, if you like, go all native. And you can likewise ignore native and opt for WebAssembly.

Sound cool? Indeed it is! But it gets better, let’s look at some of the targets:

  • Microsoft Windows
  • Apple OS X
  • Apple iOS
  • Apple WatchOS
  • Android
  • Android wearables
  • Linux x86 / 64
  • Linux ARM
  • tvOS
  • WebAssembly
  • * dot-net
  • * Java

In short: Pick the language you want, pick the RTL or framework you want, pick the target you want — and start coding!

(*) dot-net and Java are not just frameworks, they are also targets since they are Virtual Machines. WebAssembly also fall under the VM category, although the virtual machine there is bolted into Chrome and Firefox (also node.js).

Some example code

Webassembly is something that interest me more than native these days. Sure I love the speed that native has to offer, but since Javascript has become “the defacto universal platform”, and since most of my work privately is done in Javascript – it seems like the obvious place to start.

Webassembly is a bit like Javascript was 10 years ago. I remember it was a bit of a shock coming from Delphi. We had just created Smart Mobile Studio, and suddenly we realized that the classes and object the browser had to offer were close to barren. We were used to the VCL after all. So my work there was basically to implement something with enough similarity to the VCL to be familiar to to Delphi developer, without wandering too far away from established JS standards.

Webassembly is roughly in the same ballpark. Webassembly is just a runtime engine. It doesn’t give you all those nice and helpful classes out of the box. You are expected to either write that yourself – or (as luck would have it) rely on what language vendors provide.

RemObjects have a lot to offer here, because their “Delphi VCL” compatibility RTL compiles just fine for Webassembly. There is no form designer though, but I haven’t used a form designer in years. I prefer to do everything in code because that’s ultimately what works when your codebase grows large enough anyways. Even my Delphi projects are done mainly as raw code, because I like to have the option to compile with Freepascal and Lazarus.

My first test code for Oxygene Pascal with Webassembly as the target is thus very bare-bone. If there is something that has bugged me to no end, it’s that bloody HTML5 canvas. It’s a powerful thing, but it’s also overkill for per-pixel operations. So I figured that a nice, ad-hoc DIB (device independent bitmap) class will do wonders.

Note: Oxygene supports pointers, even under WebAssembly (!), but out of old habit I have avoided it. I want my code to compile for all the targets, without marking a class as “unsafe” in the dot-net paradigm. So I have avoided pointers and just use offsets instead.

namespace qtxlib;

interface

type

  // in-memory pixel format
  TPixelFormat = public (
      pf8bit  = 0,  //___8 -- palette indexed
      pf15bit = 1,  //_555 -- 15 bit encoded
      pf16bit = 2,  //_565 -- 16 bit encoded
      pf24bit = 3,  //_888 -- 24 bit native
      pf32bit = 4   //888A -- 32 bit native
      );

  TPixelBuffer = public class
  private
    FPixels:  array of Byte;
    FDepthLUT: array of Integer;
    FScanLUT: array of Integer;
    FStride:  Integer;
    FWidth:   Integer;
    FHeight:  Integer;
    FBytes:   Integer;
    FFormat:  TPixelFormat;
  protected
    function  CalcStride(const Value, PixelByteSize, AlignSize: Integer): Integer;
    function  GetEmpty: Boolean;
  public
    property  Width: Integer read FWidth;
    property  Height: Integer read FHeight;
    property  Stride: Integer read FStride;
    property  &Empty: Boolean read GetEmpty;
    property  BufferSize: Integer read FBytes;
    property  PixelFormat: TPixelFormat read FFormat;
    property  Buffer[const index: Integer]: Byte read (FPixels[&index]) write (FPixels[&index]);

    function  OffsetForPixel(const dx, dy: Integer): Integer;
    procedure Alloc(NewWidth, NewHeight: Integer; const PxFormat: TPixelFormat);
    procedure Release();

    function Read(Offset: Integer; ByteLength: Integer): array of Byte;
    procedure Write(Offset: Integer; const Data: array of Byte);

    constructor Create; virtual;

    finalizer;
    begin
      if not GetEmpty() then
        Release();
    end;
end;

TColorMixer = public class
end;

TPainter = public class
private
  FBuffer:    TPixelBuffer;
public
  property    PixelBuffer: TPixelBuffer read FBuffer;

  constructor Create(const PxBuffer: TPixelBuffer); virtual;
end;

implementation

//##################################################################################
// TPainter
//##################################################################################

constructor TPainter.Create(const PxBuffer: TPixelBuffer);
begin
  inherited Create();
  if PxBuffer  nil then
    FBuffer := PxBuffer
  else
    raise new Exception("Pixelbuffer cannot be NIL error");
end;

//##################################################################################
// TPixelBuffer
//##################################################################################

constructor TPixelBuffer.Create;
begin
  inherited Create();
  FDepthLUT := [1, 2, 2, 3, 4];
end;

function TPixelBuffer.GetEmpty: Boolean;
begin
  result := length(FPixels) = 0;
end;

function TPixelBuffer.OffsetForPixel(const dx, dy: integer): Integer;
begin
  if length(FPixels) > 0 then
  begin
    result := dy * FStride;
    inc(result, dx * FDepthLUT[FFormat]);
  end;
end;

procedure TPixelBuffer.Write(Offset: Integer; const Data: array of Byte);
begin
  for each el in Data do
  begin
    FPixels[Offset] := el;
    inc(Offset);
  end;
end;

function TPixelBuffer.Read(Offset: Integer; ByteLength: Integer): array of Byte;
begin
  result := new Byte[ByteLength];
  var xOff := 0;
  while ByteLength > 0 do
  begin
    result[xOff] := FPixels[Offset];
    dec(ByteLength);
    inc(Offset);
    inc(xOff);
  end;
end;

procedure TPixelBuffer.Alloc(NewWidth, NewHeight: Integer; const PxFormat: TPixelFormat);
begin
  if not GetEmpty() then
    Release();

  if NewWidth < 1 then
    raise new Exception("Invalid width error");

  if NewHeight  0 then
    result := ( (Result + AlignSize) - xFetch );
end;

end.

This code is just meant to give you a feel for the dialect. I have used a lot of “Delphi style” coding here, so chances are you will hardly see any difference bar namespaces and a funny looking property declaration.

Stay tuned for more posts as I explore the different aspects of Oxygene and webassembly in the days to come 🙂

RemObjects Remoting SDK?

June 3, 2019 Leave a comment

Reading this you could be forgiven for thinking that I must promote RemObjects products, It’s my job now right? Well yes, but also no.

dataabstract-illustration-rework-ro-1100The thing is, I’m really not “traveling salesman” material by any stretch of the imagination. My tolerance for bullshit is ridiculously low, and being practical of nature I loath fancy products that cost a fortune yet deliver nothing but superficial fluff.

The reasons I went to work at RemObjects are many, but most of all it’s because I have been an avid supporter of their products since they launched. I have used and seen their products in action under intense pressure, and I have come to put some faith in their solutions.

Trying to describe what it’s like to write servers that should handle thousands of active user “with or without” RemObjects Remoting SDK is exhausting, because you end up sounding like a fanatic. Having said that, I feel comfortable talking about the products because I speak from experience.

I will try to outline some of the benefits here, but you really should check it out yourself. You can download a trial directly here: https://www.remotingsdk.com/ro/

Remoting framework, what’s that?

RemObjects Remoting framework (or “RemObjects SDK” as it was called earlier) is a framework for writing large-scale RPC (remote procedure call) servers and services. Unlike the typical solutions available for Delphi and C++ builder, including those from Embarcadero I might add, RemObjects framework stands out because it distinguishes between transport, host and message-format – and above all, it’s sheer quality and ease of use.

compo

RemObjects Remoting SDK ships with a rich selection of channels and message formats

This separation between transport, host and message-format makes a lot of sense, because the parameters and data involved in calling a server-method, shouldn’t really be affected by how it got there.

And this is where the fun begins because the framework offers you a great deal of different server types (channels) and you can put together some interesting combinations by just dragging and dropping components.

How about JSON over email? Or XML over pipes?

The whole idea here is that you don’t have to just work with one standard (and pay through the nose for the privilege). You can mix and match from a rich palette of transport mediums and message-formats and instead focus on your job; to deliver a kick-ass product.

And should you need something special that isn’t covered by the existing components, inheriting out your own channel or message classes is likewise a breeze. For example, Andre Mussche have some additional components on GitHub that adds a WebSocket server and client. So there is a lot of room for expanding and building on the foundation provided by RemObjects.

And this is where RemObjects has the biggest edge (imho), namely that their solutions shaves weeks if not months off your development time. And the central aspect of that is their integrated service designer.

Integration into the Delphi IDE

Dropping components on a form is all good and well, but the moment you start coding services that deploy complex data-types (records or structures) the amount of boilerplate code can become overwhelming.

The whole point of a remoting framework is that it should expose your services to the world. Someone working in .net or Java on the other side of the planet should be able to connect, consume and invoke your services. And for that to happen every minute detail of your service has to follow standards.

61855462_10156255129755906_1396051777802993664_o

The RemObjects Service Builder integrates directly into the Delphi IDE

When you install RemObjects SDK, it also integrates into the Delphi IDE. And one of the features it integrates is a complete, separate service designer. The designer can also be used outside of the Delphi IDE, but I cannot underline enough how handy it is to be able to design your services visually, right there and then, in the Delphi IDE.

This designer doesn’t just help you design your service description (RemObjects has their own RODL file-format, which is a bit like a Microsoft WSDL file), the core purpose is to auto-generate all the boilerplate code for you — directly into your Delphi project (!)

So instead of you having to spend a week typing boilerplate code for your killer solution, you get to focus on implementing the actual methods (which is what you are supposed to be doing in the first place).

DLL services, code re-use and multi-tenancy

The idea of multi-tenancy is an interesting one. One that I talked about with regards to Rad-Server both in Oslo and London before christmas. But Rad-Server is not the only system that allows for multi-tenancy. I was doing multi-tenancy with RemObjects SDK some 14 years ago (if not earlier).

Remember how I said the framework distinguishes between transport, message and host? That last bit, namely host, is going to change how you write applications.

When you install the framework, it registers a series of custom project types inside the Delphi IDE. So if you want to create a brand new RemObjects SDK server project, you can just do that via the ordinary File->New->Other menu option.

One of the project types is called a DLL Server. Which literally means you get to isolate a whole service library inside a single DLL file! You can then load in this DLL file and call the functions from other projects. And that is, ultimately, the fundamental principle for multi-tenancy.

And no, you don’t have to compile your project with external packages for this to work. The term “dll-server” can also be a bit confusing, because we are not compiling a network server into a DLL file, we are placing the code for a service into a DLL file. I used this project type to isolate common code, so I wouldn’t have to copy unit-files all over the place when delivering the same functionality.

It’s also a great way to save money. Don’t want to pay for that new upgrade? Happy with the database components you have? Isolate them in a DLL-Server and continue to use the code from your new Delphi edition. I have Delphi XE3 Database components running inside a RemObjects DLL-Server that I use from Delphi XE 10.3.

project_types

DLL server is awesome and elegantly solves real-life problems out of the box

In my example I was doing business-logic for our biggest customers. Each of them used the same database, but they way they registered data was different. The company I worked for had bought up these projects (and thus their customers with them), and in order to keep the customers happy we couldn’t force them to re-code their systems to match ours. So we had to come up with a way to upgrade our technology without forcing a change on them.

The first thing I did was to create a “DLL server” that dealt with the database. It exposed methods like openTable(), createInvoice(), getInvoiceById() and so on. All the functions I would need to work with the data without getting my fingers dirty with SQL outside the DLL. So all the nitty gritty of SQL components, queries and whatnot was neatly isolated in that DLL file.

I then created separate DLL-Server projects for each customer, implemented their service interfaces identical to their older API. These DLL’s directly referenced the database library for authentication and doing the actual work.

62174838_10156255134895906_9195165500563259392_n

When integrated with the IDE, you are greeted with a nice welcome window when you start Delphi. Here you can open examples or check out the documentation

Finally, I wrapped it all up in a traditional Windows system service, which contained two different server-channels and the message formats they needed. When the service was started it would simply load in the DLL’s and manually register their services and types with the central channel — and voila, it worked like a charm!

Rock solid

Some 10 years after I delivered the RemObjects based solution outlined above, I got a call from my old employer. They had been victim of a devastating cyber attack. I got a bit anxious as he went on and on about damages and costs, fearing that I had somehow contributed to the situation.

targets

But it turned out he called to congratulate me! Out of all the services in their server-park, mine were the only ones left standing when the dust settled.

The RemObjects payload balancer had correctly dealt with both DDOS and brute force attacks, and the hackers were left wanting at the gates.

New job, new office, new adventures

May 12, 2019 5 comments

It’s been roughly 4 weeks since I posted a status report on Amibian.js. I normally keep people up-to-date on facebook (the “Amiga Disrupt” and also “Delphi Developer” groups). It’s been a very hectic month so I fully understand that people are asking. So let’s look at where the project is at and where we are on the time-line.

For those that might not know, I decided to leave Embarcadero a couple of months ago. I will be working out may before I move on. I wanted to write about that myself in a clean fashion, but sadly the news broke on Facebook prematurely.

Long story short, I have been very fortunate to work at Embarcadero. I am not leaving because there is anything wrong or something like that. I was hired as SC for the EMEA regions, which basically made me the support and presenter for most of europe, parts of asia and the middle east. It’s been a great adventure, but ultimately I had to admit that my passion is coding and community work. Sales is a very important part of any company, but it’s not really my cup of tea; my passion has always been research and development.

So, come first of June and I start in a new position at RemObjects. A company that has deep roots with Delphi and C++ builder users – and a company that continues to produce a wealth of high-quality, high-performance frameworks for Delphi and C++ builder. RemObjects also has a strong focus on modern languages, and have a strong portfolio of new and exciting compilers and languages to offer. The Oxygene compiler should be no stranger to Delphi developers, a powerful object-pascal dialect that can target a variety of platforms and chipsets.

Since compiler technology and run-time systems has been my main focus for well over a decade now, I feel RemObjects is a better match.

Quartex Components

Quartex Components has been an officially registered Norwegian company for a while now, so perhaps not news. What is news is that it’s now directly connected with the development of the Quartex Media Desktop (codename “Amibian.js”). While Amibian.js is an open source endeavour, there will be both free and commercial products running on top of that platform. I have written at length about Cloud Forge in the past, so I wont re-hash that again. But 2020 will see a paradigm shift in how teams and companies approach software development.

quartex

Company logo professionally milled and on its way to my new office

I will also, once there is more time, continue to sell and support software license components.

Quartex Media Desktop

The “Amibian.js” project is moving along nicely. The deadline is Q4 2019, but im hoping to wrap up the core functionality before that. So we are on track and kicking ass 🙂

amibian_01

More and more elaborate functionality is being implemented for the desktop

Here is an overview of work done this month:

  • TSystemService application type has been created (node.js)
    • TApplication now holds IPC functions (inter process communication)
    • Running child processes + sending messages is now simplicity itself
    • Database drivers are 90% done. Delete() and DeleteTable() functionality needs to be implemented in a uniform way
  • Authentication is now a separate service
    • Service database layer is finished (using SQLite3 driver by default)
    • Authentication protocol has been designed
    • Server protocol and JSON message envelopes are done
    • Presently working on the client interface
  • LDEF bytecode assembler has been improved
    • Faster symbolic lookup
    • Smarter register recognition
    • Early support for stack-frames
    • Fixed bug in parser (comma-list parse)
  • QTX framework has seen a lot of work
    • Large parts of the RTL sub-strata has been implemented
    • UTF16 codec implemented
    • QTX versions of common controls:
      • TQTXButton
      • TQTXLabel
      • TQTXToolbar
        • TQTXToolButton
        • TQTXToolSeparator
        • TQTXToolElement
      • TQTXPanel
      • TQTXCheckBox
      • .. and much, much more
  • Desktop changes
    • Link Maker functionality has been added
    • Handshake process between desktop and child app now runs on a separate timer, ensuring better conformity and a more robust initialization
    • The Quartex Editor control has been optimized
      • All redraw calls are now synchronized
      • Canvas is created on demand, avoids flicker during initial redraw
      • Support for DEL key + behavior
      • Gutter is now rendered to an offscreen bitmap and blitted into the control’s canvas. The gutter is only fully rendered when cursor forces the view to change

I will continue to keep everyone up to date about the project. As you can understand, its a bit hectic right now so please be patient – it is turning into an EPIC environment!

Understanding a stack

May 9, 2019 Leave a comment

The concept of stacks is an old one, and together with linked-lists and queues – these form the most fundamental programming concepts a developer needs to master.

But, the stack most people use today in languages like object pascal and C++ are not actual stacks; they are more like “conveniently repurposed lists“. Not a huge issue I agree, but the misconception is enough to cause confusion when people dive into low-level programming.

Adventures in assembly-land

stackishIt might seem odd to focus on something as trivial as a stack, but I have my reasons. A friend of mine who is a brilliant coder with plenty of large projects behind him recently decided to have a go at assembly coding. He was doing fine and everything was great, until he started pushing and popping  things off the stack.

After a little chat I realized that the problem was not his code, but rather how he viewed the stack. He was used to high-level versions of stacks, which in most cases are just lists storing arbitrary sized data – so he was looking at the stack as a TList<item> expecting similar behavior. Superficially a real-stack and a list-stack work the same if all you do is clean push and pop operations, but the moment you start designing a stack-scheme and push more elaborate constructs (stack-frames), things can go wrong really fast.

The nature of a real stack

A “real” stack that is a part of a hardware SOC (system on a chip) has nothing to do with lists. It’s actually a solid chunk of memory with a register to keep track of an offset into this memory block.

Let’s for sake of argument say you have 4k of stack space right? It’s clean and contains nothing, so the SP (stack pointer, or offset) is zero. What happens when you push something to the stack? for example:

push EDX

The code above simply writes the content of the EDX register to whatever offset the SP contains. It then updates the SP with the size of the data (EDX is a 32bit register, so the SP is incremented by a longword or 4 bytes). In Delphi pseudocode what happens is something like:

var LAddr := FStackBuffer;
inc(LAddr, SP);
PLongword(LAddr)^ := EDX;
inc(SP, SizeOf(EDX));

The thing about a stack is that it doesn’t manage data-length for you. And that is a big difference to remember. It will push or pop data based on the size of the source (in this case the 32bit EDX register) you use.

If you push 1024 bytes of data to a list based stack, the list keeps track of the size and data for you. So when you pop the data from the stack, you get back that data regardless. But a “real” stack couldn’t care less — which is also why it’s so easy to screw up an entire program if you make a mistake.

In short: The length of what you push – must be matched when you pop the data back (!) If you push a longword, you MUST pop a longword later.

Benefits of a real stack

call stackThe benefit is that the cost of storing values on a stack is almost zero in terms of cpu operations. A list based stack is more expensive; it will allocate memory for a record-item to hold the information about the data, then it will allocate memory to hold the actual data (depends on the type naturally) and finally copy the data into the newly allocated buffer. Hundreds if not thousands of instructions can be involved here.

A real stack will just write whatever you pushed directly into the stack-memory at whatever offset SP is at. Once written it will add the length of the write to the SP – and that’s it! So it’s one of the oldest and fastest mechanisms for lining up data in a predictable way.

Again the rules are simple: when you pop something off the stack, the size must match whatever you used to push it there. So if you pushed a longword (EDX) you also have to make sure you use a 32-bit target when you pop the value back. If you use RDX, which is 64 bit then you will essentially steal 4 bytes from something else using that stack – and all hell will break loose down the line.

Stack schemes and frames

Im not going to dig too deeply into stack-frames here, but instead write a few words about stack-schemes and using the stack to persist data your code relies on. The lines blur between those two topics anyways.

The SP (stack pointer) is not just a simple offset you can read, you can also write and change it (it also serves as a pointer). You can also read from whatever memory the SP is pointing at without polling any data from the stack.

What language developers usually do, is that they design entire structures on the stack that are, when you get into the nitty-gritty, “offset based records”. For example, lets say you have a record that looks like this:

type
PMyRecord ) ^TMyRecord;
TMyRecord = record
  first: Pointer;
  second: integer;
  Third: array[0..255] of longword;
end;

Instead of allocating conventional ram to hold that record, people push it to the stack and then use offsets to read and update the values there. A bit like a super global variable if you like. This is why when you disassemble code, you find stuff like:

mov EDX, (SP)+4

If the above record was on the stack, that pseudo code would move the field “second” into the EDX register. Because that field is 4 bytes from the stack start (providing SP points to zero).

Every programming language has a stack scheme to keep track of things. Local variables, global variables, class instances, type RTTI — most of these things are allocated in conventional ram – but there is a “program record” on the stack that makes it easy to access that information quickly.

This “moving a whole record onto the stack” is basically what a stack-frame is all about. It used to be a very costly affair with a heavy cpu speed penalty. If you look in your Delphi compiler options you will see that there is a checkbox regarding this very topic. Delphi can be told to avoid stack-frames and do register allocation instead, which was super quick compared to stack-frames – but CPU’s today are largely optimized for stack-frame allocation as default, so I doubt there is much to gain by this in 2019.

Note: A stack frame is much more, but its out of scope for this post. Google it for more info.

To sum up

When doing high-level coding you don’t really need to bother with the nuances between a TStack<item> and a “real” stack. But if you plan on digging deeper and learning a few lines of assembly – learning the differences is imperative. Its boring stuff but as fundamental as wheels on a bicycle. There is no way to avoid it, so might as well jump in.

In its absolute raw form, here is roughly the same functionality for Delphi. This was written on the fly in 2 minutes while on the road, so its purely to give you a rough idea of behavior. I would add a secondary field to keep track of the end (next insertion point), that way SP can be changed without overwriting data on new pushes.

And yes, wrapping this in a TObject utterly defeats the purpose of low-level coding, but hopefully it gives you some idea of the differences 🙂

stack_01

stack_02

Delphi AST, XML and weekend experiments

April 29, 2019 1 comment

One of the benefits of the Delphi IDE is that it’s a very rich eco-system that component writers and technology partners can tap into for their own products. I know that writing your own components is not something everyone enjoy, but knowing that you can in-fact write tools that expands the IDE using just Delphi or C++ builder, opens up for some interesting tools.

Ye old compiler bible

Ye old compiler bible

Delphi has a long tradition of “IDE enhancement” software and elaborate third-party tools that automate or delivers some benefit right in the environment. RemObjects SDK is probably the best example of how flexible the IDE truly is. RemObjects SDK integrates a whole service designer, which will generate source-code for you, update the code if you change something – and even generate service manifests for you.

There are also other tools that show off the flexibility of the IDE, ranging from code migration to advanced code refactoring and optimization.

It was with the last bit, namely code refactoring, that a third-party open-source library received a lot of deserving attention a couple of years back. A package called DelphiAST. This is a low-level syntax parser that reads Delphi source-code, applies fundamental syntax checks, and transforms the code into XML. A wet dream for anyone interested in writing advanced tooling that operates directly on source-code level.

Delphi AST

Like mentioned above, DelphiAST is a parser. Its job is very simple: parse the code, perform language level syntax checking, and convert each aspect of the code to a valid XML element. We are not talking about stuffing source-code into a CDATA segment here, but rather breaking each statement into separate tags (begin, end, if, procedure, param) so you can apply filtering, transformations and everything XML has to offer.

Back when Roman first started on DelphiAST, I got thinking — could we follow this idea further, and apply XML transformation to produce something more interesting? Would it actually be possible to approach the notion of compiling from a whole new angle? Perhaps convert between languages in a more effective way?

The short answer is: yes, everything is possible. But as always there are caveats and obstacles to overcome.

First of all, DelphiAST despite its name doesn’t actually generate a fully functional abstract symbol tree (AST). It generates a data model that is very suitable for AST generation, but not an actual AST. Everything in a programming language that can be referenced, like a method, a class, a global variable, a local variable, a parameter – are all called “symbols”. And before you can even think about processing the code, a fast and reliable AST must be in place.

Who cares?

Before I continue, you might be wondering why re-inventing the wheel is even a thing here? Why would anyone research compilers in 2019 when the world is abundant with compilers for a multitude of languages?

Because the world of computing is about to be hit by a tsunami, that’s why.

Quartex Pascal

Quartex Pascal

In the next 8-10 years the world of computing will be turned on its head. NVIDIA and roughly 100 tech companies have invested in open-source CPU designs, making it very clear that playing by Intel’s rules and bleeding royalties will no longer be tolerated. IBM has woken up from its “patent induced slumber” and is set to push their P9 cpu architecture, targeting both the high-end server and embedded market (see my article last year on PPC). At the same time Microsoft and Apple have both signaled that they are moving to ARM (an estimate of 5 years is probably reasonable). Laptop beta’s are said to be already rolling, with a commercial version expected Q3 this year (I think it wont arrive before xmas, but who knows).

Intel has remained somewhat silent about any long-term plans, but everyone that keeps an eye on hardware knows they are working like mad on next-gen FPGA. A tech that has the potential to disrupt the whole industry. Work is also being done to bridge FPGA coding with traditional code; there is no way of predicting the outcome of that though.

Oh and AMD is usurping the Intel marketshare at a steady rate — so we are in for a fight to the death.

The rise of C/C++

Those that keep tabs on languages have no doubt noticed the spike in C/C++ popularity lately. And the cause of this is that developers are safeguarding themselves for the storm to come.  C as a language might not be the most beautiful out there, but truth be told, it’s tooling requires the least amount of work to target a new platform. When a new architecture is released, C/C++ is always the first language available. You wont see C#, Flutter or Rust shipping with the latest and greatest; It’s always GCC or Clang.

Note: GCC is not just C, it’s actually a family of languages, so ironically, Gnu Basic hits a platform at the same time.

Those that have followed my blog for the past 10 years, should be more than aware of my experiments. From compiling to Javascript, generating bytecodes – and right now, moving the whole development paradigm to the browser. Hopefully my readers also recognize why this is important.

But to make you understand why I am so passionate about my compiler experiments, let’s do a little thought experiment:

Rethinking tooling

Let’s say we take Delphi, implement a bytecode format and streamline the RTL to be platform agnostic. What would the consequences of that be?

Well, first of all the compiler process would be split in two. The traditional compilation process would still be there, but it would generate bytecodes rather than machine code. That part would be isolated in a completely separate process; a process that, just like with the Delphi IDE’s infrastructure, could be outsourced to component-writers and technology partners. This in turn would provide the community with a high degree of safety, since the community itself could approach new targets without waiting for Embarcadero.

Even more, such an architecture would not be limited to machine-code. There is no law that says “you must convert bytecodes to machine code”. Since C/C++ is the foundation that modern operating-systems rest on, generating C/C++ source-code that can be built by existing compilers is a valid strategy.

There is also another factor to include in all of this, and that is Linux. Borland was correct in their assessment of Linux (the Kylix project), but they failed miserably with regards to timing. They also gravely underestimated Linux user’s sense of quality, depending on Wine (a Windows virtualization framework) to even function. They also underestimated Freepascal and Lazarus, because Linux is something FPC does exceptionally well. Competing financially against free products wont work unless you bring outstanding abilities to the table. And Linux have development tools that rival Visual Studio in quality, yet costs nothing.

But no matter how financially tricky Linux might be, we have reached the point in time where Linux is becoming mainstream. 10 years ago I had to setup my own Linux machine. There were no retailers locally that shipped a Linux box. Today I can walk into two major chains and pick dedicated Linux machines. Ubuntu in particular is well established and delivers LTS.

So for me personally, compiler tech has never been more important. And even more important is the tooling being universal and unbound by any specific API or cpu instruction-set. Firemonkey is absolutely a step in the right direction, but I think it’s a disaster to focus on native UI’s beyond a system level binding. Because replicating the same level of support and functionality for ARM, P9, RISC 5 and whatever monstrosity Intel comes up with through FPGA will take forever.

Transformation based conversion

We have wandered far off topic now, so let’s bring it back to this weekends experiment.

In short, XML transformations to convert code does work, but the right tooling have to be there to make it viable. I implemented a poor-man’s symbol table, just collecting classes, types and methods – and yeah, works just fine. What worries me a bit though is the XML parser. Microsoft has put a lot of money into XML file handling on enterprise level. When working with massive XML files (read: gigabytes) you really can’t be bothered to load the file into conventional ram and then old-school traverse the XML character by character. Microsoft operates with pure memory mapping so that you can process gigabytes like they were megabytes — but sadly, there is nothing similar for Linux, Unix or Android, that abruptly ends the fascination for me.

The only place I see using XML transformations to process source-code, is when converting to another language on source-level.

So the idea, although technically sound, gives zero benefits over the traditional process. I am however very interested in using DelphiAST to analyze and convert Delphi code directly from the IDE. But that will have to be an experiment for 2020, im booked 24/7 with Quartex Media Desktop right now.

But it was great fun playing around with DelphiAST! I loved how clean and neat the codebase has become. So if you need to work with source-code, DelphiAST is just the ticket!

Edit: You dont have to emit the code as XML. DelphiAST is perfectly happy to act as a clean parser, just saying.

TTween library for Delphi now free

March 23, 2019 5 comments

I have asked for financial backing while creating libraries that people want and enjoy, and as promised they are released into open-source land afterwards.

HexLicense was open-sourced a while back, and this time it’s TTween library that is going back to the community.

Tweening?

You have probably noticed how mobile phone UI’s have smooth movements? like on iOS when you click “back” the whole display slides smoothly into view; or that elements move, grow and shrink using fancy, accelerated effects?

This type of animation is called tweening. And the TTween Library makes it super easy to do the same for your VCL applications.

tweeners

Check out this Youtube video to see how you can make your VCL apps scale their controls more smoothly

You can fork the project here: https://bitbucket.org/cipher_diaz/ttween/src/master/

To install the system as ordinary components, just open the “Tweening.dproj” file and install as normal. Remember to add the directory to your libraries path!

Support the cause

If you like my articles and want to see more libraries and techniques, then consider donating to the project here: https://www.paypal.me/quartexNOR

paypal

Those that donate $50 or more automatically get access to the Quartex Web OS repositories, including full access to the QTX replacement RTL (for DWScript and Smart Mobile Studio).

Thank you for your support, projects like Amibian.js and the Quartex Web OS would not exist without my backers!

Building a Delphi Database engine, part four

March 23, 2019 Leave a comment

This article is over six months late (gasp!). Work at Embarcadero have been extremely time consuming, and my free time has been bound up in my ex-patreon project. So that’s why I was unable to finish in a more predictable fashion.

But better late than never — and we have finally reached one of the more exciting steps in the evolution of our database engine design, namely the place where we link our metadata to actual data.

So far we have been busy with the underlying mechanisms, how to split up larger pieces of data, how to collect these pieces and re-assemble them, how to grow and scale the database file and so on.

We ended our last article with a working persistence layer, meaning that the codebase is now able to write the metadata to itself, read it back when you open the database, persist sequences (records) – and our humble API is now rich enough to handle tasks like scaling. At the present we only support growth, but we can add file compacting later.

Tables and records

In our last article’s code, the metadata exposed a Table class. This table-class in turn exposed an interface to our field-definitions, so that we have a way to define how a table should look before we create the database.

You have probably taken a look at the code (I hope so, or much of this won’t make much sense) and noticed that the record class (TDbLibRecord) is used both as a blueprint for a table (field definitions), as well as the actual class that holds the values.

If you look at the class again (TDbLibRecord can be found in the file dblib.records.pas), you will notice that it has a series of interfaces attached to it:

  • IDbLibFields
  • IStreamPersist

The first one, which we expose in our Table as the FieldDefs property, simply exposes functions for adding and working with the fields. While somewhat different from Delphi’s traditional TFieldDefinition class, it’s familiar enough. I don’t think anyone who has used Delphi with databases would be confused around it’s members:

  IDbLibFields = interface
    ['{0D6A9FE2-24D2-42AE-A343-E65F18409FA2}']
    function    IndexOf(FieldName: string):  integer;
    function    ObjectOf(FieldName: string): TDbLibRecordField;

    function    Add(const FieldName: string; const FieldClass: TDbLibRecordFieldClass): TDbLibRecordField;
    function    Addinteger(const FieldName: string): TDbLibFieldInteger;
    function    AddStr(const FieldName: string): TDbLibFieldString;
    function    Addbyte(const FieldName: string): TDbLibFieldbyte;
    function    AddBool(const FieldName: string): TDbLibFieldboolean;
    function    AddCurrency(const FieldName: string): TDbLibFieldCurrency;
    function    AddData(const FieldName: string): TDbLibFieldData;
    function    AddDateTime(const FieldName: string): TDbLibFieldDateTime;
    function    AddDouble(const FieldName: string): TDbLibFieldDouble;
    function    AddGUID(const FieldName: string):  TDbLibFieldGUID;
    function    AddInt64(const FieldName: string): TDbLibFieldInt64;
    function    AddLong(const FieldName: string): TDbLibFieldLong;
  end;

But, as you can see, this interface is just a small part of what the class is actually about. The class can indeed hold a list of fields, each with its own datatype – but it can also persist these fields to a stream and read them back again. You can also read and write a value to each field. So it is, for all means and purposes, a single record in class form.

The term people use for this type of class is: property bag, and it was a part of the Microsoft standard components (Active X / COM) for ages. Its probably still there, but I prefer my own take on the system.

In this article we are going to finish that work, namely the ability to define a table, create a database based on the metadata, insert a new record, read records, and push the resulting binary data to the database file. And since the persistency is already in place, opening the database and reading the record back is pretty straight forward.

So this is where the metadata stops being just a blue-print, and becomes something tangible and real.

Who owns what?

Before we continue, we have to stop and think about ownership. Right now the database file persists a global list of sequences. The database class itself has no interest in who owns each sequence, if a sequence belongs to a table, if it contains a picture, a number or whatever the content might be — it simply keeps track of where each sequence begins.

So the first order of the day is to expand the metadata for tables to manage whatever records belongs to that table. In short, the database class will focus on data within its scope, and the table instances will maintain their own overview.

So the metadata suddenly need to save a list of longwords with each table. You might say that this is wasteful, that the list maintained by the database should be eliminated and that each table should keep track of it’s own data. And while that is tempting to do, there is also something to be said about maintenance. Being able to deal with persisted data without getting involved with the nitty-gritty of tables is going to be useful when things like database compacting enters at the end of our tutorial.

Locking mechanism

Delphi has a very user-friendly locking mechanism when it comes to databases. A table or dataset is either in read, edit or insert mode – and various functions are allowed or prohibited depending on that state. And it would probably be wise to merge the engine with Delphi’s own TDatabase and TTable at some point – but right now im more interested in keeping things clean and simple.

When I write “locking mechanism” I am not referring to a file-lock, or memory lock. Had we used memory-mapped files the locking mechanism would have been more elaborate. What I mean with a lock, is basically placing a table in one of the states I mentioned above. The table needs to know what exactly you want to do. Are you adding a record? Are you editing an existing record? The table code needs to know this to safely bring you from one mode to the next.

Suddenly, you realize why each table needs that extra list, because how is the table going to allow methods like first, next, last and previous? The record-list dealt with by the database is just a generic, non-ordered ledger of sequences (a global scope list if you will). Are you going to read all records back when you open the database to figure out who owns what?

A call to First() will mean a completely different offset for each table. And the logical way to handle this, is to give each table it’s own cursor. A class that keeps track of what records belongs to the table, and also keeps track of whatever states the table is in.

The database cursor

Since we are not up against Oracle or MSSQL here, but exploring database theory, I have kept the cursor as simple as I possibly could. It is a humble class that looks like this:

db_cursor

The idea of-course is that the table defaults to “read” mode, meaning that you can navigate around, record by record, or jump to a specific record using the traditional RecNo property.

The moment you want to insert or edit a record, you call the Lock() method, passing along the locking you need (edit or insert). You can then either cancel the operation or call post() to push the data down to the file.

The Lock() method is a function (bool), making it easier to write code, as such:

  if Database.Table.Cursor.Lock(cmInsert) then
  begin
    with Database.GetTableByName('access_log').cursor do
    begin
      Fields.WriteInt('id', FUserId);
      Fields.WriteStr('name', FuserName);
      Fields.WriteDateTime('access', Now);
      Post();
    end;
  end else
  raise exception.create('failed to insert record');

Im sure the are better designs, and the classes and layout can absolutely be made better; but for our purposes it should be more than adequate.

Reloading record data

In the previous articles we focused on writing data. Basically taking a stream or a buffer, breaking it into pages, and then storing the pages (or blocks) around the file where there was available space.

We cleverly crafted the blocks so that they would contain the offset to the next block in a sequence, making it possible to read back a whole sequence of blocks by just knowing the first one (!)

A part of what the cursor does is also to read data back. Whenever the RecNo field changes, meaning that you are moving around the table-records using the typical Next(), Previous(), First() etc functions — if the cursor is in read mode (meaning: you are not inserting data, nor are you editing an existing record), you have to read the record into memory. Otherwise the in-memory fields wont contain the data for that record.

Creating a cursor

One note before you dive into the code: You have to create a cursor before you can use it! So just creating a table etc wont be enough. Here is how you go about doing this:db_cursor_create

Creating the cursor will be neatly tucked into a function for the table instance, we still have other issues to deal with.

What to expect next?

Next time we will be looking at editing a record, commiting changes and deleting records. And with that in place we have finally reached the point where we can add more elaborate functionality, starting with expression parsing and filters!

You can check out the code here: https://bitbucket.org/cipher_diaz/dbproject/src/master/

Support the cause

If you like my articles and want to see more libraries and techniques, then consider donating to the project here: https://www.paypal.me/quartexNOR

paypal

Those that donate $50 or more automatically get access to the Quartex Web OS repositories, including full access to the QTX replacement RTL (for DWScript and Smart Mobile Studio).

Thank you for your support, projects like Amibian.js and the Quartex Web OS would not exist without my backers!

/Jon

VMWare: A Delphi developers best friend

March 3, 2019 1 comment

Full disclosure: I am not affiliated with any particular virtualization vendor of any sorts. The reason I picked VMWare was because their product was faster when I compared the various solutions. So feel free to replace the word VMWare with whatever virtualization software suits your needs.

On Delphi Developer we get new members and questions about Delphi and C++ builder every day. It’s grown into an awesome community where we help each other, do business, find jobs and even become personal friends.

A part of what we do in our community, is to tip each other about cool stuff. It doesn’t have to be directly bound to Delphi or code either; people have posted open source graphic programs, video editing, database designers – as long as its open source or freeware its a great thing (we have a strict policy of no piracy or illegal copying).

Today we got talking about VMWare and how its a great time saver. So here goes:

Virtualization

Virtualization is, simply put, a form of emulation. Back in the mid 90s emulators became hugely popular because for the first time in history – we had CPU’s powerful enough to emulate other computers at full speed. This was radical because up until that point, you needed special hardware to do that. You had also been limited to emulating legacy systems with no practical business value.

vmware

VmWare Workstation is an amazing piece of engineering

Emulation has always been there, even back in the 80s with 16 bit computers. But while it was technically possible, it was more a curiosity than something an office environment would benefit from (unless you used expensive compute boards). We had to wait until the late 90s to see commercial-grade x86 emulation hitting the market, with Virtuozzo releasing Parallels in 1997 and VMWare showing up around 1998. Both of these companies grew out of the data-center culture and academia.

It’s also worth noting that modern CPU’s now support virtualization on  hardware level, so when you are “virtualizing” Windows the machine code is not interpreted or JIT compiled – it runs on the same CPU as your real system.

Why does it matter

Virtualization is not just for data-centers and server-farms, it’s also for desktop use. My personal choice was VMWare because I felt their product performed better than the others. But in all fairness it’s been a few years since I compared between systems, so that might be different today.

53145702_10156048129355906_2019146241329332224_o

A screengrab of my desktop, here showing 3 virtual machines running. I have 64 gigabyte memory and these 3 virtual machines consume around 24 gigabytes and uses 17% of the Intel i7 CPU power during compile. It hardly registers on the CPU stats when idle.

VMWare Workstation is a desktop application available for Windows, Linux and OS X. And it allows me to create virtual machines, or “emulations” if you like. The result is that I can run multiple instances of Windows on a single PC. The virtual machines are all sandbox in large hard-disk files, and you have to install Windows or Linux into these virtual systems.

The bonus though is fantastic. Once you have installed an operating-system, you can copy it, move it, do partial cloning (only changes are isolated in new sandboxes) and much, much more. The cloning functionality is incredibly powerful, especially for a developer.

It also gives you something called snap-shot support. A snapshot is, like the word hints to, a copy of whatever state your virtual-machine is in at that point in time. This is a wonderful feature if you remember to use it properly. I try to take snapshots before I install anything, be it larger systems like Delphi, or just utility applications I download. Should something go wrong with the tools your work depends on — you can just roll back to a previous snapshot (!)

A great time saver

Updates to development tools are always awesome, but there are times when things can go wrong. But if you remember to take a snapshot before you install a program, or before you install a component package — should something go wrong, then rolling back to a clean point is reduced to a mouse click.

I mean, imagine you update your development tools right? Suddenly you realize that a component package your software depends on doesn’t work. If you have installed your devtools directly on the metal, you suddenly have a lot of time-consuming work to do:

  • Re-install your older devtools
  • Re-install your components and fix broken paths

That wont be a problem if you only have 2-3 packages, but I have hundreds of components install on my rig. Just getting my components working can take almost a full work-day, and I’m not exaggerating (!).

With VMWare, I just roll back to when all was fine, and go about my work like nothing happened.

I made a quick, slapdash video to demonstrate how easy VmWare makes my Delphi and JS development. If you are not using virtualization I hope this video at least makes it a bit clearer why so many do.

vmware_youtube

Click the image to watch the video on YouTube

Repository updates

February 25, 2019 2 comments

As most know by now, I was running a successful campaign on Patreon until recently. I know that some are happy with Patreon, but hopefully my experience will be a wakeup call about the total lack of rights you as a creator have – should Patreon decide they don’t understand what you are doing (which I can only presume was the case, because I was never given a reason at all). You can read more about my experience with Patreon by clicking here.

Setting up repositories

Having to manually build a package for each tier that I have backers for would be a disaster. It was time-consuming and repetitive enough to create packages on Patreon, and I don’t have time to reverse engineer Patreon either. Which I might do in the future and release as open-source just to give them a kick in the groin back.

To make it easier for my backers to get the code they want, I have isolated each project and sub-project in separate repositories on BitBucket. This covers Delphi, Smart Pascal, LDEF and everything else.

cloud_ripper

The CloudRipper architecture is coming along nicely. Here running on ODroid XU4

I’m just going to continue with the Tiers I originally made on Patreon, and use my blog as the news-center for everything. Since I tend to blog about things from a personal point of view, be it for Delphi, JavaScript or Smart Pascal — I doubt people will notice the difference.

So far the following repositories have been setup:

  • Amibian.js Server (Quartex Web OS)
  • Amibian.js Client
  • HexLicense
  • TextCraft (source-code parser for Delphi and Smart Pascal)
  • UAE.js (a fork of SAE, the JS implementation of UAE)

I need to clean up the server repository a bit, because right now it contains both the server-code and various sub projects. The LDEF assembler program for example, is also under that repository — and it belongs in its own repository as a unique sub-project.

The following repositories will be setup shortly:

  • Tweening library for Delphi and Smart Pascal
  • PixelRage graphics library
  • ByteRage bugger library
  • LDEF (containing both Delphi and Smart Pascal code)
  • LDEF Assembler

It’s been extremely busy days lately so I need to do some thinking about how we can best organize things. But rest assured that everyone that backs the project, or a particular tier, will get access to what they support.

Support and backing

I have been looking at various ways to do this, but since most backers have just said they want Paypal, I decided to go for that. So donations can be done directly via paypal. One of the new features in Paypal is repeated payments, so setting up a backer-plan should be easy enough. I am notified whenever someone gives a donation, so it’s pretty easy to follow-up on.

 

 

Updates used to be monthly, but with the changes they will be ad-hoc, meaning that I will commit directly. I do have local backups and a local git server, so for parts of the project the commits will be issued at the end of each month.

While all support is awesome, here are the tiers I used on Patreon:

  • $5 – “high-five”, im not a coder but I support the cause
  • $10 – Tweening animation library
  • $25 – License management and serial minting components
  • $35 – Rage libraries: 2 libraries for fast graphics and memory management
  • $45 – LDef assembler, virtual machine and debugger
  • $50 – Amibian.js (pre compiled) and Ragnarok client / server library
  • $100 – Amibian.js binaries, source and setup
  • $100+ All the above and pre-made disk images for ODroid XU4 and x86 on completion of the Amibian.js project (12 month timeline).

So to back the project like before, all you do is:

  1. Register with Bitbucket (free user account)
  2. Setup donation and inform me of your Bitbucket user-name
  3. I add you on BitBucket so you are granted access rights

Easy. Fast and reliable.

The QTX RTL

Those that have been following the Amibian.js project might have noticed that a fair bit of QTX units have appeared in the code? QTX is a run-time library compatible with Smart Mobile Studio and DWScript. Eventually the code that makes up Amibian.js will become a whole new RTL. This RTL has nothing to do with Smart Mobile Studio and ships with its own license.

Amigian_display

QTX approaches the DOM in more efficient way. Its faster, smaller and more powerful

Backers at $45 or beyond access to this code automatically. If you use Smart Mobile Studio then this is a must. It introduces a ton of classes that doesn’t exist in Smart Pascal, and also introduces a much faster and clean visual component framework.

If you want to develop visual applications using QTX and DWScript,  then that is OK,  providing the license is respected (LGPL, non commercial use).

Well, stay tuned for more info and news!

Leaving Patreon: Developers be warned

February 17, 2019 4 comments

As a person I’m quite optimistic. I like to think the glass is half-full rather than half-empty. I have spent over a decade building up a thriving Delphi and C++ builder community on social media, I have built up a rich creative community for node and JavaScript on the side — not to mention retro computing, embedded tech and IOT. For better or for worse I think most developers in the Embarcadero camp have heard my name or engage in one of the 12 groups I manage around the world on a daily basis. It’s been hard work but man, it’s been worth every minute. We have so much fun and I get to meet awesome coders on a daily basis. It’s become an intrinsic part of my life.

I have been extremely fortunate in that despite my disadvantage, a spine injury in 2012 – not to mention being situated in Norway rather than the united states; despite these obstacles to overcome I work for a great American company, and I get to socialize and have friends all over the planet.

The global village is the concept, or philosophy, that technology makes it possible no-matter where you live, to connect and be a part of something bigger. You don’t have to be a startup in the san-francisco area to work with the latest tech. Sure a commute from Burlingame to Redwood beats a 14 hour flight from Norway any day of the week — but that’s the whole idea: we have Skype now, and Slack and Github; you don’t have to physically be on location to be a part of a great company. The only requirement is that you make yourself relevant to your field of expertise.

Patreon, a digital talent agency

Patreon is a service that grew straight out of the global village. If the world is just one place, one great big family of human beings with great ideas, then where is the digital stage that helps nurturing these individuals? I mean, you can have a genius kid living in poverty in Timbuktu that could crack a mathematical problem on the other side of the globe. The next musical prodigy could be living in a loft in Germany, but his or her voice will never be heard unless it’s recognized and given positive feedback.

“The irony is that Patreon doesn’t even pass their own safety tests. That should make you think twice about their operation”

My examples are extremes I agree, most people on Patreon are like me, creative but absolutely not cracking math problems for Nasa; nor am I singing a duet with Bono any time soon. But that’s the fun thing about the world – namely that all things have value when put in the correct context. Life is about combinations, and you just have to find one that works for you.

village

The global village, the idea of unity through diversity

The global village is this wonderful idea that we can use technology to transcend the limitations the world oppose on us, be they nationality, color, gender or location. Good solutions know no bounds and manifests wherever a mind welcomes it. Perhaps a somewhat romantic idea, if not naive, but it seems the only reasonable solution given the rapid changes we face as a species.

In my case, I love to make software components in my spare time. My day job is packed and I couldn’t squeeze in more work during the weekdays if I wanted to, so I only have a couple of hours after-work and the weekends to “do my thing”. So being a total geek I relax by making components. Some play chess, the guitar or whatever — I relax by coding something useful.

Obviously “code components” are completely useless to anyone who is not a software developer. The relevance is further clipped by the programming-language they are written for, and ultimately the functionality they provide. Patreon for me was a way to finance the evolution of these components. A way of self motivating myself to keep them up to date and available.

I also put a larger project on Patreon, namely the cloud desktop system people know as “Amibian.js” or “Quartex Web OS”. Amibian being the nickname, or codename.

Patreon seemed like the perfect match. I could take these seemingly unrelated topics, Delphi and C++ builder specific components and a cloud architecture, and assign each component and project to separate “tiers” that the audience could pick from. This was great! People could now subscribe to the tier’s they wanted, and would be notified whenever there was an update or new features. And I could respond to service messages in one place.

The Tier System

The thing about software is that it’s not maintained on infinite repeat. You don’t fix a component that is working. And you don’t issue updates unless you have fixed bugs or added new functionality. A software subscription secures a customer access to all and any updates, with a guarantee of X number of updates a year. And equally important, that they can get help if they are stuck.

“when you are shut down without so much as an explanation, with nothing but positive feedback, zero refunds and over 1682 people actively following the progress — that is utterly unacceptable behavior”

I set a relatively low number of guaranteed updates per year for the components (4). The things that would see the most updates were the Rage Libraries (PixelRage and ByteRage) and Amibian.js, but not until Q3 when all the modules would come together as a greater whole — something my backers are aware of and have never had a problem with.

Amigian_display

Amibian.js running on ODroid XU4, a $45 single board computer

The tiers I ended up with was:

  • $5 – “high-five”, im not a coder but I support the cause
  • $10 – Tweening animation library
  • $25 – License management and serial minting components
  • $35 – Rage libraries: 2 libraries for fast graphics and memory management
  • $45 – LDef assembler, virtual machine and debugger
  • $50 – Amibian.js (pre compiled) and Ragnarok client / server library
  • $100 – Amibian.js binaries, source and setup
  • $100+ All the above and pre-made disk images for ODroid XU4 and x86 on completion of the Amibian.js project (12 month timeline).

Note: Each tier covers everything before them. So if you pick the $35 tier, that also includes access to the license management system and the animation library.

As you can see, the tier-system that is intrinsic to Patreon, solves the software subscription model elegantly. After all, it would be unreasonable to demand $100 a month for a small component like the Tweening library. A programmer that just needs that library and nothing else shouldnt have to pay for anything else.

Here is a visual representation, showing graphically why my tiers are organized as they are, and how they all fit into a greater whole:

tier_dependencies

The server-side aspect of the architecture would take days to document, but a general overview of the micro-service architecture is fairly easy to understand:

tier_dependencies2

Each of the tiers were picked because they represent key aspects of what we need to create a visually pleasing, fast and reliable, distributed (each part running on separate machines or boards) cloud eco-system. Supporters can just get the parts they need, or support the bigger project. Everyone get’s what they want – all is well.

The thing some people don’t grasp, is that you are not getting something to just put on Amazon or Azure, you are getting your own Amazon or Azure – with source code! You are not getting services, you are getting the actual code that allows YOU to set up your own services. Anyone with a server can become a service provider and offer both hosting and software access. And they can expand on this without having to ask permission or pay through the nose.

So it’s a little bit bigger than first meets the eye.

I Move In Mysterious Ways ..

Roughly 3 weeks ago I was busy preparing the monthly updates.

Since each tier is separate but also covers everything before it (like explained above) I have to prepare a set of inclusive updates. The good news is that I only have to do this once and then add it as an attachment to my posts. Once added I can check of all the backers in that tier. I don’t have to manually email each backer, physically copy my songs or creations onto CD and send it – we live in the digital age as members of the global village. Or so i thought.

So I published two of the minor cases first: the full HTML5 assembly program, that can be run both inside Amibian.js as a hosted application — or as a solo program directly in the browser. So here people can write machine-code in the browser, assemble it to bytecodes, run the code, inspect registers, disassemble the bytecodes and all the normal stuff you expect from an assembler.

This update was special because the program contained the IPC (inter process communication) layer that developers use to make their programs talk to the desktop. So for developers looking to make their own web programs access the filesystem, open dialogs (normal system features), that code was quite important to get!

tier_posts

Although published, none of my backers could see them due to the suspended status

The second post was a free addition, the QTX library which is an open-source RTL (run time library) compatible with the Smart Pascal Compiler. While not critical at this juncture, several of my backers use Smart Mobile Studio, and for them to get access to a whole new RTL that can be used for open-source, is very valuable indeed.

I was just about to compress the Amibian.js source-code and binaries when I got a message on Facebook by a backer:

“Dude, your Patreon is shut down, what is happening?”

What? hang on let me check i replied, and rushed into Patreon where the following header greeted me:

tier_header

What the hell Patreon? I figured there must be some misunderstanding and that perhaps I missed an email or something that needed attention. I get close to 50 emails a day (literally) so it does happen that I miss one. I also check my spam folder regularly in case my google filters have been careless and flagged a serious email as spam. But there was nothing. Not a word.

Ok, so let’s check the page feedback, has there been any complaints? Perhaps a backer has misunderstood something and I need to clear that up? But nope. I had nothing but positive feedback and not even a single refund request.  In fact the Amibian.js group on Facebook has grown to 1,662 members. Which shows that the project itself holds considerable interest outside software development circles.

Well, let’s get on this quickly I thought, so I rushed off an email asking why Patreon would do such a thing? My entire Patreon page was visibly marked with the above banner, so my backers never even saw the updates I had issued.

Instead, the impression people would get, was that I was involved in something so devious that it demanded my account to be suspended. Talk about shooting first and asking later. I have never in my life seen such behavior from a company anywhere, especially not in the united states; Americans don’t take kindly to companies behaving like bullies.

Just Contact Support, If You Can Find Them

To make a long story short it took over a week before Patreon replied to my emails. I sent a total of 3 emails asking what on earth would have prompted them to shut down a successful campaign. And how they found it necessary to slander the project without even informing me of the problem. Surely a phone call could have sorted this up in minutes? Where I come from you pick up the phone or get in contact with people before you flag them in public.

patreon

Sounds great, sadly it’s pure fiction

The response I got was that “some mysterious activity had been reported on my page”, and that they wanted my name, address, phone number and credit card (4 last digits). Which I found funny because with the exception of credit-card details, I always put my name, address, phone numbers and email etc. at the head of my letters.

I’m not a 16-year-old kid working out of a garage, im a 46-year-old established software developer that have worked as a professional for close to 3 decades. Unlike the present generation I moved into my first apartment when I was 16, and was working as an author for various tech magazines by the time I was 17. I also finished college at the same time and went on to higher-education (2 years electrical engineering, 3 years arts and media, six years at the university in oslo, followed by 4 years of computer science and then certifications). The focus being, that Patreon is used to dealing with young creators that will go along with things that grown men would not accept.

But what really piss me off, was that they never even bothered to explain what this “mysterious behavior” actually was? I write about code, clustering, Delphi, JavaScript and bytecodes for christ sake. I might have published updates and code wearing a hoodie at one point, in a darken room, listening to Enigma.. but honestly: there is not enough mystery in my life to cover an episode of Scooby-Doo.

Either way, I provided the information they wanted and expected the problem to be resolved asap. Two days at themost. Maybe three, but that was pushing it.

It’s now close to 3 weeks since this ridiculous temporary suspension occurred, and neither have I been given any explanation to what I have done, nor have they removed the ban on the content. I must have read their guidelines 100 times by now, but given the nature of their ruling (which are more than reasonable), I can’t see that I have violated a single one:

  • No pornography and adult content
  • No hate speech against minorities or forms of religious extremism
  • No piracy or spreading copyrighted material
  • No stealing from backers

Let’s go over them one by one shall we?

Pornography and adult content

Seriously? I don’t have time to loaf around glaring at naked women (i’m a geek, I look weird enough as it is), and after 46 years on this planet I know what a woman looks like nude from every possible angle; I don’t need to run around like a retard posting pictures of body parts. And if you are talking about me — good lord is there a marked for hobbits? Surely the world has enough on it’s plate. Sorry, never been huge on porn.

And for the record, porn is for teenagers and singles. The moment you love someone deeply, the moment you have children together — it changes you profoundly. You get a bond to your wife or girlfriend that makes you not want to be with others. Not all men are into smut, some of us are invested more deeply in a relationship.

Hate speech and religious extremism

Hm, that’s a tough one (sigh). Did you know that one of my best friends is so gay – that he began to speculated that he actually was a liquid? He makes me laugh so bad and he’s probably the best human being I have ever met. I actually went with him on Pride last year, not because i’m gay but because he needed someone to hold the other side of the banner. That’s what friends do. Besides, I looked awesome, what can I say.

As for religion I am a registered Tibetan Buddhist. I believe in fluffy pillows, comfy robes, mother nature and quite frankly I find the world inside us far more interesting than the mess outside. You cant be extreme in Buddhism: “Be kind now, or ill hug you until you weep the tears of compassion!”. Buddhism sucks as an extreme doctrine.

So I’m going to go out on a limb and say nuuuu to both.

Piracy and copyrighted material

Eh, I’m kinda writing the software from scratch before your eyes (including the run-time-library for the compiler), so as far as worthy challenges go, piracy would be the opposite. I am a huge fan of classical operating-systems though, like the Amiga; But unlike most people I actually took the time to ask permission to use a OS4 inspired CSS theme-file.

asana

The Amibian.js project is well organized and I have worked systematically through a well planned architecture. This is not some slap-dash project made for a quick buck

Most people just create a theme-file and don’t bother to ask. I did, and Trevor Dickinson was totally cool about it. And not a single byte has been taken or stolen from anyone. The default theme file is inspired by Amiga OS 4.1, but the thing is: the icons are all freeware. Mason, the guy that did the OS icons, have released large sets of icons into GPL. There is also a website called OS4Depot where people publish icons and backdrops that are free for all.

So if this “mysterious activity” is me posting a picture of a picture (not a typo) of an obscure yet loved operating-system, rest assured that it’s not violating anyone.

Stealing from backers

That they even include this as a point is just monumental. Patreon is a service established to make that impossible (sigh); meaning that the time-frame where you deliver updates or whatever – and the time when the payout is delivered, that is the window where backers can file a complaint or demand a refund.

And yes, complaints on fraud would indeed (and should!) flag the account as potentially dubious — but again, I have not a single complaint. Not even a refund request, which I believe is pretty uncommon.

And even if this was the case, shutting down an account without so much as a dialog in 2019? Who the hell becomes a thief for 600 dollars? Im not some kid in a garage, I make twice that a day as a consultant in Oslo, why the heck would I setup a public account in the US, only to run off with 600 bucks! I have standing offers for projects continuously, I havent applied for a job since the 90s – so if I needed some extra money I would have taken a side project.

I even posted to let my backers know I had a cold last month just to make sure everyone knew in case I was unavailable for a couple of days. Truly the tell-tell sign of a criminal mastermind if I ever saw one ..

tier_refunds

Sorry Patreon, but your behavior is unacceptable

Hopefully your experience with Patreon has not been like mine. They spent somewhere in the range of 5 weeks just to register me, while friends of mine in the US was up and running in less than 2 days.

We are now 3 weeks into a temporary suspension, which means that most of my backers will run out of patience and just leave. It sends a signal of being whimsical about other people’s trust, and that people take a risk if they back my project.

At this point it doesn’t matter that none of these thoughts are true, because they are thoughts that anyone would think when a project remains flagged for so long.

What should scare you as a creator with Patreon though, is that they can do this to anyone. There is nothing you can do, neither to prove your innocence or sort out a misunderstanding — because you are not even told what you allegedly have done wrong. I also find it alarming that Patreon actually doesn’t have a phone-number listed, nor do they have offices you can call or reach out to.

The irony is that Patreon doesn’t even pass their own safety tests. That should make you think twice about their operation. I had heard the rumors about them, but I honestly did not believe a company could operate like this in our day and age. Especially not in the united states. It undermines the whole spirit of US as a technological hub. No wonder people are setting up shop in China instead, if this is how they are treated in the valley.

After this long, and the damage they have caused, I have no option than to inform my backers to terminate their pledges. I will have to relocate my project to a host that has more experience with software development, and who treats human beings with common decency and respect.

If I by accident had violated any of their guidelines, although I cannot see how I could have, I have no problem taking responsibility. But when you are shut down without so much as an explanation, with nothing but positive feedback, zero refunds and over 1682 people actively following the progress — that is utterly unacceptable.

It is a great shame. Patreon symbolized, for a short time, that the global village had matured into more than an idea. But I categorically refuse to be treated like this and find their modus-operandi insulting.

Stay Well Clear

If you as a developer have a chance to set up shop elsewhere, then I urge you to do so. And make sure your host have common infrastructure such as a phone number. Patreon have taken the art of avoiding direct contact to a whole new level. It is absolutely mind-boggling.

I honestly don’t think Patreon understands software development at all. Many have voiced more sinister motives for my shutdown, since the project obviously is a threat to various companies. But I don’t believe in conspiracies. Although, if Patreon does this to enough creators on interval, the interest rates from holding the assets would be substantial.

It could be that the popularity of the project grew so fast that it was picked up as a statistical anomaly, but surely that should be a good thing? Not to mention a potential case study Patreon could have used as a success story? I mean, Amibian.js didn’t get up and running until october, so stopping a project 5 months into a 12 month timeline makes absolutely no sense. Unless someone did this on purpose.

Either way, this has been a terrible experience and I truly hope Patreon get’s their act together. They could have resolved this with a phone-call, yet chose to let it fester for almost a month.

Their loss.