Archive
Using multiple languages is the same project
Most compilers can only handle a single syntax for any project, but the Elements compiler from RemObjects deals with 5 (five!) different languages -even within the same project. That’s pretty awesome and opens up for some considerable savings.
I mean, it’s not always easy to find developers for a single language, but when you can approach your codebase from C#, Java, Go, Swift and Oxygene (object pascal) at the same time (inside the same project even!), you suddenly have some options. Especially since you can pick exotic targets like WebAssembly. Or what about compiling Java to .net bytecodes? Or using the VCL from C#? It’s pretty awesome stuff!
Check out Marc Hoffmans article on the Elements compiler toolchain and how you can mix and match between languages, picking the best from each — while still compiling to a single binary of llvm optimized code:
Click on the picture to be redirected
Check out RemObjects Remoting SDK
RemObjects Remoting SDK is one of those component packages that have become more than the sum of it’s part. Just like project Jedi has become standard equipment almost, Remoting SDK is a system that all Delphi and Freepascal developers should have in their toolbox.
In this article I’m going to present the SDK in broad strokes; from a viewpoint of someone who haven’t used the SDK before. There are still a large number of Delphi developers that don’t know it even exists – hopefully this post will shed some light on why the system is worth every penny and what it can do for you.
I should also add, that this is a personal blog. This is not an official RemObjects presentation, but a piece written by me based on my subjective experience and notions. We have a lot of running dialog at Delphi Developer on Facebook, so if I read overly harsh on a subject, that is my personal view as a Delphi Developer.
Stop re-inventing the wheel
Delphi has always been a great tool for writing system services. It has accumulated a vast ecosystem of non-visual components over the years, both commercial and non-commercial, and this allows developers to quickly aggregate and expose complex behavior — everything from graphics processing to databases, file processing to networking.
The challenge for Delphi is that writing large composite systems, where you have more than a single service doing work in concert, is not factored into the RTL or project type. Delphi provides a bare-bone project type for system services, and that’s it. Depending on how you look at it, it’s either a blessing or a curse. You essentially start on C level.
So fundamental things like IPC (inter process communication) is something you have to deal with yourself. If you want multi-tenancy that is likewise not supported out of the box. And all of this is before we venture into protocol standards, message formats and async vs synchronous execution.
The idea behind Remoting SDK is to get away from this style of low-level hacking. Without sounding negative, it provides the missing pieces that Delphi lacks, including the stuff that C# developers enjoy under .net (and then some). So if you are a Delphi developer who look over at C# with smudge of envy, then you are going to love Remoting SDK.
Say goodbye to boilerplate mistakes
Writing distributed servers and services is boring work. For each function you expose, you have to define the parameters and data-types in a portable way, then you have to implement the code that represents the exposed function and finally the interface itself that can be consumed by clients. The latter must be defined in a way that works with other languages too, not just Delphi. So while server tech in it’s essential form is quite simple, it’s the infrastructure that sets the stage of how quickly you can apply improvements and adapt to change.
For example, let’s say you have implemented a wonderful new service. It exposes 60 awesome functions that your customers can consume in their own work. The amount of boilerplate code for 60 distributed functions, especially if you operate with composite data types, is horrendous. It is a nightmare to manage and opens up for sloppy, unnecessary mistakes.
After you install Remoting SDK, the service designer becomes a part of the IDE
This is where Remoting SDK truly shines. When you install the software, it integrates it’s editors and wizards closely with the Delphi IDE. It adds a ton of new project types, components and whatnot – but the most important feature is without a doubt the service designer.
Start the service-designer in any server or service project and you can edit the methods, data types and interfaces your system expose to the world
As the name implies, the service designer allows you to visually define your services. Adding a new function is a simple click, the same goes for datatypes and structures (record types). These datatypes are exposed too and can be consumed from any modern language. So a service you make in Delphi can be used from C#, C/C++, Java, Oxygene, Swift (and visa-versa).
Auto generated code
A service designer is all good and well I hear you say, but what about that boilerplate code? Well Remoting SDK takes care of that too (kinda the point). Whenever you edit your services, the designer will auto-generate a new interface unit for you. This contains the classes and definitions that describe your service. It will also generate an implementation unit, with empty functions; you just need to fill in the blanks.
The designer is also smart enough not to remove code. So if you go in and change something, it won’t just delete the older implementation procedure. Only the params and names will be changed if you have already written some code.
Having changed a service, hitting F9 re-generates the interface code automatically. Your only job is to fill in the code for each method in the implementation units. The SDK takes care of everything else for you
The service information, including the type information, is stored in a special file format called “rodl”. This format is very close to Microsoft WSDL format, but it holds more information. It’s important to underline that you can import the service directly from your servers (optional naturally) as WSDL. So if you want to consume a Remoting SDK service using Delphi’s ordinary RIO components, that is not a problem. Visual Studio likewise imports and consumes services – so Remoting SDK behaves identical regardless of platform or language used.
Remoting SDK is not just for Delphi, just to be clear on that. If you are presently using both Delphi and C# (which is a common situation), you can buy a license for both C# and Delphi and use whatever language you feel is best for a particular task or service. You can even get Remoting SDK for Javascript and call your service-stack directly from your website if you like. So there are a lot of options for leveraging the technology.
Transport is not content
OK so Remoting SDK makes it easy to define distributed services and servers. But what about communication? Are we boxed into RemObjects way of doing things?
The remoting framework comes with a ton of components, divided into 3 primary groups:
- Servers
- Channels (clients)
- Messages
The reason for this distinction is simple: the ability to transport data, is never the same as the ability to describe data. For example, a message is always connected to a standard. It’s job is ultimately to serialize (represent) and de-serialize data according to a format. The server’s job is to receive a request and send a response. So these concepts are neatly decoupled for maximum agility.
As of writing the SDK offers the following message formats:
- Binary
- Post
- SOAP
- JSON
If you are exposing a service that will be consumed from JavaScript, throwing in a TROJSONMessage component is the way to go. If you expect messages to be posted from your website using ordinary web forms, then TROPostMessage is a perfect match. If you want XML then TROSOAPMessage rocks, and if you want fast, binary messages – well then there is TROBinaryMessage.
What you must understand is that you don’t have to pick just one! You can drop all 4 of these message formats and hook them up to your server or channel. The SDK is smart enough to recognize the format and use the correct component for serialization. So creating a distributed service that can be consumed from all major platforms is a matter of dropping components and setting a property.
If you double-click on a server or channel, you can link message components with a simple click. No messy code snippets in sight.
Multi-tenancy out of the box
With the release of Rad-Server as a part of Delphi, people have started to ask what exactly multi-tenancy is and why it matters. I have to be honest and say that yes, it does matter if you are creating a service stack where you want to isolate the logic for each customer in compartments – but the idea that this is somehow new or unique is not the case. Remoting SDK have given users multi-tenancy support for 15+ years, which is also why I haven’t been too enthusiastic with Rad-Server.
Now don’t get me wrong, I don’t have an axe to grind with Rad-Server. The only reason I mention it is because people have asked how i feel about it. The tech itself is absolutely welcome, but it’s the licensing and throwing Interbase in there that rubs me the wrong way. If it could run on SQLite3 and was free with Enterprise I would have felt different about it.
There are various models for multi-tenancy, but they revolve around the same principles
To get back on topic: multi-tenancy means that you can dynamically load services and expose them on demand. You can look at it as a form of plugin functionality. The idea in Rad-Server is that you can isolate a customer’s service in a separate package – and then load the package into your server whenever you need it.
Some of the components that ship with the system
The reason I dislike Rad-Server in this respect, is because they force you to compile with packages. So if you want to write a Rad-Server system, you have to compile your entire project as package-based, and ship a ton of .dpk files with your system. Packages is not wrong or bad per-se, but they open your system up on a fundamental level. There is nothing stopping a customer from rolling his own spoof package and potentially bypass your security.
There is also an issue with un-loading a package, where right now the package remains in memory. This means that hot-swapping packages without killing the server wont work.
Rad-Server is also hardcoded to use Interbase, which suddenly bring in licensing issues that rubs people the wrong way. Considering the price of Delphi in 2019, Rad-Server stands out as a bit of an oddity. And hardcoding a database into it, with the licensing issues that brings -just rendered the whole system mute for me. Why should I pay more to get less? Especially when I have been using multi-tenancy with RemObjects for some 15 years?
With Remoting SDK you have something called DLL servers, which does the exact same thing – but using ordinary DLL files (not packages!). You don’t have to compile your system with packages, and it takes just one line of code to make your main dispatcher aware of the loaded service.
This actually works so well that I use Remoting SDK as my primary “plugin” system. Even when I write ordinary desktop applications that has nothing to do with servers or services – I always try to compartmentalize features that could be replaced in the future.
For example, I’m a huge fan of ElevateDB, which is a native Delphi database engine that compiles directly into your executable. By isolating that inside a DLL as a service, my application is now engine agnostic – and I get a break from buying a truck load of components every time Delphi is updated.
Saving money
The thing about DLL services, is that you can save a lot of money. I’m actually using an ElevateDB license that was for Delphi 2007. I compiled the engine using D2007 into a DLL service — and then I consume that DLL from my more modern Delphi editions. I have no problem supporting or paying for components, that is right and fair, but having to buy new licenses for every single component each time Delphi is updated? This is unheard of in other languages, and I would rather ditch the platform all together than forking out $10k ever time I update.
A DLL server can be used for many things if you are creative about it
While we are on the subject – Hydra is another great money saver. It allows you to use .net and Java libraries (both visual and non-visual) with Delphi. With Hydra you can design something in .net, compile it into a DLL file, and then use that from Delphi.
But — you can also compile things from Delphi, and use it in newer versions of Delphi. Im not forking out for a Developer Express update just to use what I have already paid for in the latest Delphi. I have one license, I compile the forms and components into a Hydra Module — and then use it from newer Delphi editions.
Hydra, which is a separate product, allows you to stuff visual components and forms inside a vanilla DLL. It allows cross language use, so you can finally use Java and .net components inside your Delphi application
Bonjour support
Another feature I love is the zero configuration support. This is one of those things that you often forget, but that suddenly becomes important once you deploy a service stack on cluster level.
Remoting SDK comes with support for Apple Bonjour, so if you want to use that functionality you have to install the Bonjour library from Apple. Once installed on your host machines, your RemObjects services can find each other.
ZeroConfig is not that hard to code manually. You can roll your own using UDP or vanilla messages. But getting service discovery right can be fiddly. One thing is broadcasting an UDP message saying “here I am”, it’s something else entirely to allow service discovery on cluster level.
If Bonjour is not your cup of tea, the SDK provides a second option, which is RemObjects own zero-config hub. You can dig into the documentation to find out more about this.
What about that IPC stuff you mentioned?
I mentioned IPC (inter process communication) at the beginning here, which is a must have if you are making a service stack where each member is expected to talk to the others. In a large server-system the services might not exist on the same, physical hardware either, so you want to take height for that.
With the SDK this is just another service. It takes 10 minutes to create a DLL server with the functionality to send and receive messages – and then you just load and plug that into all your services. Done. Finished.
Interestingly, Remoting SDK supports named-pipes. So if you are running on a Windows network it’s even easier. Personally I prefer to use a vanilla TCP/IP based server and channel, that way I can make use of my Linux blades too.
Building on the system
There is nothing stopping you from expanding the system that RemObjects have established. You are not forced to only use their server types, message types and class framework. You can mix and match as you see fit – and also inherit out your own variation if you need something special.
For example, WebSocket is an emerging standard that has become wildly popular. Remoting SDK does not support that out of the box, the reason is that the standard is practically identical to the RemObjects super-server, and partly because there must be room for third party vendors.
Andre Mussche took the time to implement a WebSocket server for Remoting SDK a few years back. Demonstrating in the process just how easy it is to build on the existing infrastructure. If you are already using Remoting SDK or want WebSocket support, head over to his github repository and grab the code there: https://github.com/andremussche/DelphiWebsockets
I could probably write a whole book covering this framework. For the past 15 years, RemObjects Remoting SDK is the first product I install after Delphi. It has become standard for me and remains an integral part of my toolkit. Other packages have come and gone, but this one remains.
Hopefully this post has tickled your interest in the product. No matter if you are maintaining a legacy service stack, or thinking about re implementing your existing system in something future-proof, this framework will make your life much, much easier. And it wont break the bank either.
You can visit the product page here: https://www.remotingsdk.com/ro/default.aspx
And you can check out the documentation here: https://docs.remotingsdk.com/
A Delphi propertybag
A long, long time ago, way back in the previous century, I often had to adjust a Visual Basic project my company maintained. Going from object-pascal to VB was more than a little debilitating; Visual Basic was not a compiled language like Delphi is, and it lacked more or less every feature you needed to produce good software.
I could probably make a VB clone using Delphi pretty easily. But I think the world has experienced enough suffering, no need to add more evil to the universe
Having said that, I have always been a huge fan of Basic (it was my first language after all, it’s what schools taught in the 70s and 80s). I think it was a terrible mistake for Microsoft to retire Basic as a language, because it’s a great way to teach kids the fundamentals of programming.
Visual Basic is still there though, available for the .Net framework, but to call it Basic is an insult of the likes of GFA Basic, Amos Basic and Blitz Basic; the mighty compilers of the past. If you enjoyed basic before Microsoft pushed out the monstrosity that is Visual Basic, then perhaps swing by GitHub and pick up a copy of BlitzBasic? BlitzBasic is a completely different beast. It compiles to machine-code, allows inline assembly, and has been wildly popular for game developers over the years.
A property bag
The only feature that I found somewhat useful in Visual Basic, was an object called a propertybag. It’s just a fancy name for a dictionary, but it had a couple of redeeming factors beyond lookup ability. Like being able to load name-value-pairs from a string, recognizing datatypes and exposing type-aware read/write methods. Nothing fancy but handy when dealing with database connection-strings, shell parameters and the like.
So you could feed it strings like this:
first=12;second=hello there;third=3.14
And the class would parse out the names and values, stuff it in a dictionary, and you could easily extract the data you needed. Nothing fancy, but handy on rare occasions.
A Delphi version
Im mostly porting code from Delphi to Oxygene these days, but here is my Delphi implementation of the propertybag object. Please note that I haven’t bothered to implement the propertybag available in .Net. The Delphi version below is based on the Visual Basic 6 version, with some dependency injection thrown in for good measure.
unit fslib.params;
interface
{.$DEFINE SUPPORT_URI_ENCODING}
uses
System.SysUtils,
System.Classes,
Generics.Collections;
type
(* Exceptions *)
EPropertybag = class(exception);
EPropertybagReadError = class(EPropertybag);
EPropertybagWriteError = class(EPropertybag);
EPropertybagParseError = class(EPropertybag);
(* Datatypes *)
TPropertyBagDictionary = TDictionary ;
IPropertyElement = interface
['{C6C937DF-50FA-4984-BA6F-EBB0B367D3F3}']
function GetAsInt: integer;
procedure SetAsInt(const Value: integer);
function GetAsString: string;
procedure SetAsString(const Value: string);
function GetAsBool: boolean;
procedure SetAsBool(const Value: boolean);
function GetAsFloat: double;
procedure SetAsFloat(const Value: double);
function GetEmpty: boolean;
property Empty: boolean read GetEmpty;
property AsFloat: double read GetAsFloat write SetAsFloat;
property AsBoolean: boolean read GetAsBool write SetAsBool;
property AsInteger: integer read GetAsInt write SetAsInt;
property AsString: string read GetAsString write SetAsString;
end;
TPropertyBag = Class(TInterfacedObject)
strict private
FLUT: TPropertyBagDictionary;
strict protected
procedure Parse(NameValuePairs: string);
public
function Read(Name: string): IPropertyElement;
function Write(Name: string; Value: string): IPropertyElement;
procedure SaveToStream(const Stream: TStream);
procedure LoadFromStream(const Stream: TStream);
function ToString: string; override;
procedure Clear; virtual;
constructor Create(NameValuePairs: string); virtual;
destructor Destroy; override;
end;
implementation
{$IFDEF SUPPORT_URI_ENCODING}
uses
system.NetEncoding;
{$ENDIF}
const
cnt_err_sourceparameters_parse =
'Failed to parse input, invalid or damaged text error [%s]';
cnt_err_sourceparameters_write_id =
'Write failed, invalid or empty identifier error';
cnt_err_sourceparameters_read_id =
'Read failed, invalid or empty identifier error';
type
TPropertyElement = class(TInterfacedObject, IPropertyElement)
strict private
FName: string;
FData: string;
FStorage: TPropertyBagDictionary;
strict protected
function GetEmpty: boolean; inline;
function GetAsInt: integer; inline;
procedure SetAsInt(const Value: integer); inline;
function GetAsString: string; inline;
procedure SetAsString(const Value: string); inline;
function GetAsBool: boolean; inline;
procedure SetAsBool(const Value: boolean); inline;
function GetAsFloat: double; inline;
procedure SetAsFloat(const Value: double); inline;
public
property AsFloat: double read GetAsFloat write SetAsFloat;
property AsBoolean: boolean read GetAsBool write SetAsBool;
property AsInteger: integer read GetAsInt write SetAsInt;
property AsString: string read GetAsString write SetAsString;
property Empty: boolean read GetEmpty;
constructor Create(const Storage: TPropertyBagDictionary; Name: string; Data: string); overload; virtual;
constructor Create(Data: string); overload; virtual;
end;
//#############################################################################
// TPropertyElement
//#############################################################################
constructor TPropertyElement.Create(Data: string);
begin
inherited Create;
FData := Data.Trim();
end;
constructor TPropertyElement.Create(const Storage: TPropertyBagDictionary;
Name: string; Data: string);
begin
inherited Create;
FStorage := Storage;
FName := Name.Trim().ToLower();
FData := Data.Trim();
end;
function TPropertyElement.GetEmpty: boolean;
begin
result := FData.Length < 1;
end;
function TPropertyElement.GetAsString: string;
begin
result := FData;
end;
procedure TPropertyElement.SetAsString(const Value: string);
begin
if Value FData then
begin
FData := Value;
if FName.Length > 0 then
begin
if FStorage nil then
FStorage.AddOrSetValue(FName, Value);
end;
end;
end;
function TPropertyElement.GetAsBool: boolean;
begin
TryStrToBool(FData, result);
end;
procedure TPropertyElement.SetAsBool(const Value: boolean);
begin
FData := BoolToStr(Value, true);
if FName.Length > 0 then
begin
if FStorage nil then
FStorage.AddOrSetValue(FName, FData);
end;
end;
function TPropertyElement.GetAsFloat: double;
begin
TryStrToFloat(FData, result);
end;
procedure TPropertyElement.SetAsFloat(const Value: double);
begin
FData := FloatToStr(Value);
if FName.Length > 0 then
begin
if FStorage nil then
FStorage.AddOrSetValue(FName, FData);
end;
end;
function TPropertyElement.GetAsInt: integer;
begin
TryStrToInt(FData, Result);
end;
procedure TPropertyElement.SetAsInt(const Value: integer);
begin
FData := IntToStr(Value);
if FName.Length > 0 then
begin
if FStorage nil then
FStorage.AddOrSetValue(FName, FData);
end;
end;
//#############################################################################
// TPropertyBag
//#############################################################################
constructor TPropertyBag.Create(NameValuePairs: string);
begin
inherited Create;
FLUT := TDictionary.Create();
NameValuePairs := NameValuePairs.Trim();
if NameValuePairs.Length > 0 then
Parse(NameValuePairs);
end;
destructor TPropertyBag.Destroy;
begin
FLut.Free;
inherited;
end;
procedure TPropertyBag.Clear;
begin
FLut.Clear;
end;
procedure TPropertyBag.Parse(NameValuePairs: string);
var
LList: TStringList;
x: integer;
LId: string;
LValue: string;
LOriginal: string;
{$IFDEF SUPPORT_URI_ENCODING}
LPos: integer;
{$ENDIF}
begin
// Reset content
FLUT.Clear();
// Make a copy of the original text
LOriginal := NameValuePairs;
// Trim and prepare
NameValuePairs := NameValuePairs.Trim();
// Anything to work with?
if NameValuePairs.Length > 0 then
begin
{$IFDEF SUPPORT_URI_ENCODING}
// Check if the data is URL-encoded
LPos := pos('%', NameValuePairs);
if (LPos >= low(NameValuePairs) )
and (LPos 0 then
Begin
(* Populate our lookup table *)
LList := TStringList.Create;
try
LList.Delimiter := ';';
LList.StrictDelimiter := true;
LList.DelimitedText := NameValuePairs;
if LList.Count = 0 then
raise EPropertybagParseError.CreateFmt(cnt_err_sourceparameters_parse, [LOriginal]);
try
for x := 0 to LList.Count-1 do
begin
LId := LList.Names[x].Trim().ToLower();
if (LId.Length > 0) then
begin
LValue := LList.ValueFromIndex[x].Trim();
Write(LId, LValue);
end;
end;
except
on e: exception do
raise EPropertybagParseError.CreateFmt(cnt_err_sourceparameters_parse, [LOriginal]);
end;
finally
LList.Free;
end;
end;
end;
end;
function TPropertyBag.ToString: string;
var
LItem: TPair;
begin
setlength(result, 0);
for LItem in FLut do
begin
if LItem.Key.Trim().Length > 0 then
begin
result := result + Format('%s=%s;', [LItem.Key, LItem.Value]);
end;
end;
end;
procedure TPropertyBag.SaveToStream(const Stream: TStream);
var
LData: TStringStream;
begin
LData := TStringStream.Create(ToString(), TEncoding.UTF8);
try
LData.SaveToStream(Stream);
finally
LData.Free;
end;
end;
procedure TPropertyBag.LoadFromStream(const Stream: TStream);
var
LData: TStringStream;
begin
LData := TStringStream.Create('', TEncoding.UTF8);
try
LData.LoadFromStream(Stream);
Parse(LData.DataString);
finally
LData.Free;
end;
end;
function TPropertyBag.Write(Name: string; Value: string): IPropertyElement;
begin
Name := Name.Trim().ToLower();
if Name.Length > 0 then
begin
if not FLUT.ContainsKey(Name) then
FLut.Add(Name, Value);
result := TPropertyElement.Create(FLut, Name, Value) as IPropertyElement;
end else
raise EPropertybagWriteError.Create(cnt_err_sourceparameters_write_id);
end;
function TPropertyBag.Read(Name: string): IPropertyElement;
var
LData: String;
begin
Name := Name.Trim().ToLower();
if Name.Length > 0 then
begin
if FLut.TryGetValue(Name, LData) then
result := TPropertyElement.Create(LData) as IPropertyElement
else
raise EPropertybagReadError.Create(cnt_err_sourceparameters_read_id);
end else
raise EPropertybagReadError.Create(cnt_err_sourceparameters_read_id);
end;
end.
BTree for Delphi
Click here to read
A few weeks back I posted an article on RemObjects blog regarding universal code, and how you with a little bit of care can write code that easily compiled with both Oxygene, Delphi and Freepascal. With emphasis on Oxygene.
The example I used was a BTree class that I originally ported from Delphi to Smart Pascal, and then finally to Oxygene to run under WebAssembly.
Long story short I was asked if I could port the code back to Delphi in its more or less universal form. Naturally there are small differences here and there, but nothing special that distinctly separates the Delphi version from Oxygene or Smart Pascal.
Why this version?
If you google BTree and Delphi you will find loads of implementations. They all operate more or less identical, using records and pointers for optimal speed. I decided to base my version on classes for convenience, but it shouldn’t be difficult to revert that to use records if you absolutely need it.
What I like about this BTree implementation is that it’s very functional. Its easy to traverse the nodes using the ForEach() method, you can add items using a number as an identifier, but it also supports string identifiers.
I also changed the typical data reference. The data each node represent is usually a pointer. I changed this to variant to make it more functional.
Well, here is the Delphi version as promised. Happy to help.
unit btree;
interface
uses
System.Generics.Collections,
System.Sysutils,
System.Classes;
type
// BTree leaf object
TQTXBTreeNode = class(TObject)
public
Identifier: integer;
Data: variant;
Left: TQTXBTreeNode;
Right: TQTXBTreeNode;
end;
[Weak]
TQTXBTreeProcessCB = reference to procedure (const Node: TQTXBTreeNode; var Cancel: boolean);
EBTreeError = class(Exception);
TQTXBTree = class(TObject)
private
FRoot: TQTXBTreeNode;
FCurrent: TQTXBTreeNode;
protected
function GetEmpty: boolean; virtual;
function GetPackedNodes: TList;
public
property Root: TQTXBTreeNode read FRoot;
property Empty: boolean read GetEmpty;
function Add(const Ident: integer; const Data: variant): TQTXBTreeNode; overload; virtual;
function Add(const Ident: string; const Data: variant): TQTXBTreeNode; overload; virtual;
function Contains(const Ident: integer): boolean; overload; virtual;
function Contains(const Ident: string): boolean; overload; virtual;
function Remove(const Ident: integer): boolean; overload; virtual;
function Remove(const Ident: string): boolean; overload; virtual;
function Read(const Ident: integer): variant; overload; virtual;
function Read(const Ident: string): variant; overload; virtual;
procedure Write(const Ident: string; const NewData: variant); overload; virtual;
procedure Write(const Ident: integer; const NewData: variant); overload; virtual;
procedure Clear; overload; virtual;
procedure Clear(const Process: TQTXBTreeProcessCB); overload; virtual;
function ToDataArray: TList;
function Count: integer;
procedure ForEach(const Process: TQTXBTreeProcessCB);
destructor Destroy; override;
end;
implementation
//#############################################################################
// TQTXBTree
//#############################################################################
destructor TQTXBTree.Destroy;
begin
if FRoot nil then
Clear();
inherited;
end;
procedure TQTXBTree.Clear;
var
lTemp: TList;
x: integer;
begin
if FRoot nil then
begin
// pack all nodes to a linear list
lTemp := GetPackedNodes();
try
// release each node
for x := 0 to ltemp.Count-1 do
begin
lTemp[x].Free;
end;
finally
// dispose of list
lTemp.Free;
// reset pointers
FCurrent := nil;
FRoot := nil;
end;
end;
end;
procedure TQTXBTree.Clear(const Process: TQTXBTreeProcessCB);
begin
ForEach(Process);
Clear();
end;
function TQTXBTree.GetPackedNodes: TList;
var
LData: Tlist;
begin
LData := TList.Create();
ForEach( procedure (const Node: TQTXBTreeNode; var Cancel: boolean)
begin
LData.Add(Node);
Cancel := false;
end);
result := LData;
end;
function TQTXBTree.GetEmpty: boolean;
begin
result := FRoot = nil;
end;
function TQTXBTree.Count: integer;
var
LCount: integer;
begin
ForEach( procedure (const Node: TQTXBTreeNode; var Cancel: boolean)
begin
inc(LCount);
Cancel := false;
end);
result := LCount;
end;
function TQTXBTree.ToDataArray: TList;
var
Data: TList;
begin
Data := TList.Create();
ForEach( procedure (const Node: TQTXBTreeNode; var Cancel: boolean)
begin
Data.add(Node.data);
Cancel := false;
end);
result := data;
end;
function TQTXBTree.Add(const Ident: string; const Data: variant): TQTXBTreeNode;
begin
result := Add( Ident.GetHashCode(), Data);
end;
function TQTXBTree.Add(const Ident: integer; const Data: variant): TQTXBTreeNode;
var
lNode: TQTXBtreeNode;
begin
LNode := TQTXBTreeNode.Create();
LNode.Identifier := Ident;
LNode.Data := data;
if FRoot = nil then
FRoot := LNode;
FCurrent := FRoot;
while true do
begin
if (Ident FCurrent.Identifier) then
begin
if (FCurrent.right = nil) then
begin
FCurrent.right := LNode;
break;
end else
FCurrent := FCurrent.right;
end else
break;
end;
result := LNode;
end;
function TQTXBTree.Read(const Ident: string): variant;
begin
result := Read( Ident.GetHashCode() );
end;
function TQTXBTree.Read(const Ident: integer): variant;
begin
FCurrent := FRoot;
while FCurrent nil do
begin
if (Ident Fcurrent.Identifier) then
FCurrent := FCurrent.Right
else
begin
result := FCUrrent.Data;
break;
end
end;
end;
procedure TQTXBTree.Write(const Ident: string; const NewData: variant);
begin
Write( Ident.GetHashCode(), NewData);
end;
procedure TQTXBTree.Write(const Ident: integer; const NewData: variant);
begin
FCurrent := FRoot;
while (FCurrent nil) do
begin
if (Ident Fcurrent.Identifier) then
FCurrent := FCurrent.Right
else
begin
FCurrent.Data := NewData;
break;
end
end;
end;
function TQTXBTree.Contains(const Ident: string): boolean;
begin
result := Contains( Ident.GetHashCode() );
end;
function TQTXBTree.Contains(const Ident: integer): boolean;
begin
result := false;
if FRoot nil then
begin
FCurrent := FRoot;
while ( (not Result) and (FCurrent nil) ) do
begin
if (Ident Fcurrent.Identifier) then
FCurrent := FCurrent.Right
else
begin
Result := true;
break;
end
end;
end;
end;
function TQTXBTree.Remove(const Ident: string): boolean;
begin
result := Remove( Ident.GetHashCode() );
end;
function TQTXBTree.Remove(const Ident: integer): boolean;
var
LFound: boolean;
LParent: TQTXBTreeNode;
LReplacement,
LReplacementParent: TQTXBTreeNode;
LChildCount: integer;
begin
FCurrent := FRoot;
LFound := false;
LParent := nil;
LReplacement := nil;
LReplacementParent := nil;
while (not LFound) and (FCurrent nil) do
begin
if (Ident FCurrent.Identifier) then
begin
LParent := FCurrent;
FCurrent := FCurrent.right;
end else
LFound := true;
if LFound then
begin
LChildCount := 0;
if (FCurrent.left nil) then
inc(LChildCount);
if (FCurrent.right nil) then
inc(LChildCount);
if FCurrent = FRoot then
begin
case (LChildCOunt) of
0: begin
FRoot := nil;
end;
1: begin
if FCurrent.right = nil then
FRoot := FCurrent.left
else
FRoot :=FCurrent.Right;
end;
2: begin
LReplacement := FRoot.left;
while (LReplacement.right nil) do
begin
LReplacementParent := LReplacement;
LReplacement := LReplacement.right;
end;
if (LReplacementParent nil) then
begin
LReplacementParent.right := LReplacement.Left;
LReplacement.right := FRoot.Right;
LReplacement.left := FRoot.left;
end else
LReplacement.right := FRoot.right;
end;
end;
FRoot := LReplacement;
end else
begin
case LChildCount of
0: if (FCurrent.Identifier < LParent.Identifier) then
Lparent.left := nil else
LParent.right := nil;
1: if (FCurrent.Identifier < LParent.Identifier) then
begin
if (FCurrent.Left = NIL) then
LParent.left := FCurrent.Right else
LParent.Left := FCurrent.Left;
end else
begin
if (FCurrent.Left = NIL) then
LParent.right := FCurrent.Right else
LParent.right := FCurrent.Left;
end;
2: begin
LReplacement := FCurrent.left;
LReplacementParent := FCurrent;
while LReplacement.right nil do
begin
LReplacementParent := LReplacement;
LReplacement := LReplacement.right;
end;
LReplacementParent.right := LReplacement.left;
LReplacement.right := FCurrent.right;
LReplacement.left := FCurrent.left;
if (FCurrent.Identifier < LParent.Identifier) then
LParent.left := LReplacement
else
LParent.right := LReplacement;
end;
end;
end;
end;
end;
result := LFound;
end;
procedure TQTXBTree.ForEach(const Process: TQTXBTreeProcessCB);
function ProcessNode(const Node: TQTXBTreeNode): boolean;
begin
if Node nil then
begin
if Node.left nil then
begin
result := ProcessNode(Node.left);
if result then
exit;
end;
Process(Node, result);
if result then
exit;
if (Node.right nil) then
begin
result := ProcessNode(Node.right);
if result then
exit;
end;
end;
end;
begin
ProcessNode(FRoot);
end;
end.
Calling node.js from Delphi
We got a good question about how to start a node.js program from Delphi on our Facebook group today (third one in a week?). When you have been coding for years you often forget that things like this might not be immediately obvious. Hopefully I can shed some light on the options in this post.
Node or chrome?
Just to be clear: node.js has nothing to do with chrome or chromium embedded. Chrome is a web-browser, a completely visual environment and ecosystem.
Node.js is the complete opposite. It is purely a shell based environment, meaning that it’s designed to run services and servers, with emphasis on the latter.
The only thing node.js and chrome have in common, is that they both use the V8 JavaScript runtime engine to load, JIT compile and execute scripts at high speed. Beyond that, they are utterly alien to each other.
Can node.js be embedded into a Delphi program?
Technically there is nothing stopping a C/C++ developer from compiling the node.js core system as C++ builder compatible .obj files; files that can then be linked into a Delphi application through references. But this also requires a bit of scaffolding, like adding support for malloc_, free_ and a few other procedures – so that your .obj files uses the same memory manager as your Delphi code. But until someone does just that and publish it, im afraid you are stuck with two options:
- Use a library called Toby, that keeps node.js in a single DLL file. This is the most practical way if you insist on hosting your own version of node.js
- Add node.js as a prerequisite and give users the option to locate the node.exe in your application’s preferences. This is the way I would go, because you really don’t want to force users to stick with your potentially outdated or buggy build.
So yes, you can use toby and just add the toby dll file to your program folder, but I have to strongly advice against that. There is no point setting yourself up for maintaining a whole separate programming language, just because you want JavaScript support.
“How many in your company can write high quality WebAssembly modules?”
If all you want to do is support JavaScript in your application, then I would much rather install Besen into Delphi. Besen is a JavaScript runtime engine written in Freepascal. It is fully compatible with Delphi, and follows the ECMA standard to the letter. So it is extremely compatible, fast and easy to use.
Like all Delphi components Besen is compiled into your application, so you have no dependencies to worry about.
Starting a node.js script
The easiest way to start a node.js script, is to simply shell-execute out of your Delphi application. This can be done as easily as:
ShellExecute(Handle, 'open', PChar('node.exe'), pchar('script.js'), nil, SW_SHOW);
This is more than enough if you just want to start a service, server or do some work that doesn’t require that you capture the result.
If you need to capture the result, the data that your node.js program emits on stdout, there is a nice component in the Jedi Component Library. Also plenty of examples online on how to do that.
If you need even further communication, you need to look for a shell-execute that support pipes. All node.js programs have something called a message-channel in the Javascript world. In reality though, this is just a named pipe that is automatically created when your script starts (with the same moniker as the PID [process identifier]).
If you opt for the latter you have a direct, full duplex message channel directly into your node.js application. You just have to agree with yourself on a protocol so that your Delphi code understands what node.js is saying, and visa versa.
UDP or TCP
If you don’t want to get your hands dirty with named pipes and rolling your own protocol, you can just use UDP to let your Delphi application communicate with your node.js process. UDP is practically without cost since its fundamental to all networking stacks, and in your case you will be shipping messages purely between processes on localhost. Meaning: packets are never sent on the network, but rather delegated between processes on the same machine.
In that case, I suggest you ship in the port you want your UDP server to listen on, so that your node.js service acts as the server. A simple command-line statement like:
node.exe myservice.js 8090
Inside node.js you can setup an UDP server with very little fuzz:
function setupServer(port) {
var os = require("os");
var dgram = require("dgram");
var socket = dgram.createSocket("udp4");
var MULTICAST_HOST = "224.0.0.236";
var BROADCAST_HOST = "255.255.255.255";
var ALL_PORT = 60540;
var MULTICAST_TTL = 1; // Local network
socket.bind(port);
socket.on('listening', function() {
socket.setMulticastLoopback(true);
socket.setMulticastTTL(MULTICAST_TTL);
socket.addMembership(multicastHost);
if(broadcast) { socket.setBroadcast(true); }
});
socket.on('message', parseMessage);
}
function parseMessage(message, rinfo) {
try {
var messageObject = JSON.parse(message);
var eventType = messageObject.eventType;
} catch(e) {
}
}
Note: the code above assumes a JSON text message.
You can then use any Delphi UDP client to communicate with your node.js server, Indy is good, Synapse is a good library with less overhead – there are many options here.
Do I have to learn Javascript to use node.js?
If you download DWScript you can hook-up the JS-codegen library (see library folder in the DWScript repository), and use that to compile DWScript (object pascal) to kick-ass Javascript. This is the same compiler that was used in Smart Mobile Studio.
“Adding WebAssembly to your resume is going to be a hell of a lot more valuable in the years to come than C# or Java”
Another alternative is to use Freepascal, they have a pas2js project where you can compile ordinary object-pascal to javascript. Naturally there are a few things to keep in mind, both for DWScript and Freepascal – like avoiding pointers. But clean object pascal compiles just fine.
If JavaScript is not your cup of tea, or you simply don’t have time to learn the delicate nuances between the DOM (document object model, used by browsers) and the 100% package oriented approach deployed by node.js — then you can just straight up to webassembly.
RemObjects Software has a kick-ass webassembly compiler, perfect if you dont have the energy or time to learn JavaScript. As of writing this is the fastest and most powerful toolchain available. And I have tested them all.
WebAssembly, no Javascript needed
You might remember Oxygene? It used to be shipped with Delphi as a way to target Microsoft CLR (common language runtime) and the .net framework.
Since then Oxygene and the RemObjects toolchain has evolved dramatically and is now capable of a lot more than CLR support.
- You can compile to raw, llvm optimized machine code for 8 platforms
- You can compile to CLR/.Net
- You can compile to Java bytecodes
- You can compile to WebAssembly!
WebAssembly is not Javascript, it’s important to underline that. WebAssembly was created especially for developers using traditional languages, so that traditional compilers can emit web friendly, binary code. Unlike Javascript, WebAssembly is a purely binary format. Just like Delphi generates machine-code that is linked into a final executable, WebAssembly is likewise compiled, linked and emitted in binary form.
If that sounds like a sales pitch, it’s not. It’s a matter of practicality.
- WebAssembly is completely barren out of the box. The runtime environment, be it V8 for the browser or V8 for node.js, gives you nothing out of the box. You don’t even have WriteLn() to emit text.
- Google expects compiler makers to provide their own RTL functions, from the fundamental to the advanced. The only thing V8 gives you, is a barebone way of referencing objects and functions on the other side, meaning the JS and DOM world. And that’s it.
So the reason i’m talking a lot about Oxygene and RemObjects Elements (Elements is the name of the compiler toolchain RemObjects offers), is because it ships with an RTL. So you are not forced to start on actual, literal assembly level.
If you don’t want to study JavaScript, Oxygene and Elements from RemObjects is the solution
RemObjects also delivers a DelphiVCL compatibility framework. This is a clone of the Delphi VCL / Freepascal LCL. Since WebAssembly is still brand new, work is being done on this framework on a daily basis, with updates being issued all the time.
Note: The Delphi VCL framework is not just for WebAssembly. It represents a unified framework that can work anywhere. So if you switch from WebAssembly to say Android, you get the same result.
The most important part of the above, is actually not the visual stuff. I mean, having HTML5 visual controls is cool – but chances are you want to use a library like Sencha, SwiftUI or jQueryUI to compose your forms right? Which means you just want to interface with the widgets in the DOM to set and get values.
You probably want to use a fancy UI library, like jQuery UI. This works perfectly with Elements because you can reference the controls from your WebAssembly module. You dont have to create TButton, TListbox etc manually
The more interesting stuff is actually the non-visual code you get access to. Hundreds of familiar classes from the VCL, painstakingly re-created, and usable from any of the 5 languages Elements supports.
You can check it out here: https://github.com/remobjects/DelphiRTL
Skipping JavaScript all together
I dont believe in single languages. Not any more. There was a time when all you needed was Delphi and a diploma and you were set to conquer the world. But those days are long gone, and a programmer needs to be flexible and have a well stocked toolbox.
Knowing where you want to be is half the journey
The world really don’t need yet-another-c# developer. There are millions of C# developers in India alone. C# is just “so what?”. Which is also why C# jobs pays less than Delphi or node.js system service jobs.
What you want, is to learn the things others avoid. If JavaScript looks alien and you feel uneasy about the whole thing – that means you are growing as a developer. All new things are learned by venturing outside your comfort zone.
How many in your company can write high quality WebAssembly modules?
How many within one hour driving distance from your office or home are experts at WebAssembly? How many are capable of writing industrial scale, production ready system services for node.js that can scale from a single instance to 1000 instances in a large, clustered cloud environment?
Any idiot can pick up node.js and knock out a service, but with your background from Delphi or C++ builder you have a massive advantage. All those places that can throw an exception that JS devs usually ignore? As a Delphi or Oxygene developer you know better. And when you re-apply that experience under a different language, suddenly you can do stuff others cant. Which makes your skills valuable.
The Quartex Media Desktop have made even experienced node / web developers gasp. They are not used to writing custom-controls and large-scale systems, which is my advantage
So would you learn JavaScript or just skip to WebAssembly? Honestly? Learn a bit of both. You don’t have to be an expert in JavaScript to compliment WebAssembly. Just get a cheap book, like “Node.js for beginners” and “JavaScript the good parts” ($20 a piece) and that should be more than enough to cover the JS side of things.
Adding WebAssembly to your resume and having the material to prove you know your stuff, is going to be a hell of a lot more valuable in the years to come than C#, Java or Python. THAT I can guarantee you.
And, we have a wicked cool group on Facebook you can join too: Click here to visit RemObjects Developer.
Getting into Node.js from Delphi
Delphi is one of the best development toolchains for Windows. I have been an avid fan of Delphi since it was first released, and before that – Turbo Pascal too. Delphi has a healthy following – and despite popular belief, Delphi scores quite well on the Tiobe Index.
As cool and efficient as Delphi might be, there are situations where native code wont work. Or at the very least, be less efficient than the alternatives. Delphi has a broad wingspan, from low-level assembler all the way to classes and generics. But JavaScript and emerging web technology is based on a completely different philosophy, one where native code is regarded as negative since it binds you to hardware.
Getting to grips with the whole JavaScript phenomenon, be it for mobile, embedded or back-end services, can be daunting if all you know is native code. But thankfully there are alternatives that can help you become productive quickly, something I will brush over in this post.
JavaScript without JavaScript
Before we dig into the tools of the trade, I want to cover alternative ways of enjoying the power of node.js and Javascript. Namely by using compilers that can convert code from a traditional language – and emit fully working JavaScript. There are a lot more options than you think:
Quartex Media Desktop is a complete environment written purely in JavaScript. Both Server, Cluster and front-end is pure JavaScript. A good example of what can be done.
- Swift compiles for JavaScript, and Apple is doing some amazing things with the new and sexy SwiftUI tookit. If you know your way around Swift, you can compile for Javascript
- Go can likewise be compiled to JS:
- C/C++ can be compiled to asm.js courtesy of EmScripten. It uses clang to first compile your code to llvm bitcode, and then it converts that into asm.js. You have probably seen games like Quake run in the browser? That was asm.js, a kind of precursor to WebAssembly.
- NS Basic compiles for JavaScript, this is a Visual Basic 6 style environment with its own IDE even
For those coming straight from Delphi, there are a couple of options to pick from:
- Freepascal (pas2js project)
- DWScript compiles code to JavaScript, this is the same compiler that we used in Smart Pascal earlier
- Oxygene, the next generation object-pascal from RemObjects compiles to WebAssembly. This is by far the best option of them all.
I strongly urge you to have a look at Elements, here running in Visual Studio
JavaScript, Asm.js or WebAssembly?
Asm.js is by far the most misunderstood technology in the JavaScript ecosystem, so let me just cover that before we move on:
A few years back JavaScript gained support for memory buffers and typed arrays. This might not sound very exciting, but in terms of speed – the difference is tremendous. The default variable type in JavaScript is what Delphi developers know as Variant. It assumes the datatype of the values you assign to it. Needless to say, there is a lot of overhead when working with variants – so JavaScript suddenly getting proper typed arrays was a huge deal.
It was then discovered that JavaScript could manipulate these arrays and buffers at high speed, providing it only used a subset of the language. A subset that the JavaScript runtime could JIT compile more easily (turn into machine-code).
So what the EmScripten team did was to implement a bytecode based virtual-machine in Javascript, and then they compile C/C++ to bytecodes. I know, it’s a huge project, but the results speak for themselves — before WebAssembly, this was as fast as it got with JavaScript.
WebAssembly
WebAssembly is different from both vanilla JavaScript and Asm.js. First of all, it’s executed at high speed by the browser itself. Not like asm.js where these bytecodes were executed by JavaScript code.
Water is a fast, slick and platform independent IDE for Elements. The same IDE for OS X is called Fire. You can use RemObjects Elements from either Visual Studio or Water
Secondly, WebAssembly is completely JIT compiled by the browser or node.js when loading. It’s not like Asm.js where some parts are compiled, others are interpreted. WebAssembly runs at full speed and have nothing to do with traditional JavaScript. It’s actually a completely separate engine.
Out of all the options on the table, WebAssembly is the technology with the best performance.
Kits and strategies
The first thing you need to be clear about, is what you want to work with. The needs and requirements of a game developer will be very different from a system service developer.
Here are a couple of kits to think about:
- Mobile developer
- Implement your mobile applications using Oxygene, compiling for WebAssembly (Elements)
- RemObjects Remoting SDK for client / server communication
- Use Freepascal for vanilla JavaScript scaffolding when needed
- Service developer
- Implement libraries in Oxygene to benefit from the speed of WebAssembly
- Use RemObjects Data Abstract to make data-access uniform and fast
- Use Freepascal for boilerplate node.js logic
- Desktop developer
- For platform independent desktop applications, WebAssembly is the way to go. You will need some scaffolding (plain Javascript) to communicate with the application host – but the 99.9% of your code will be better under WebAssembly.
- Use Cordova / Phonegap to “bundle” your WebAssembly, HTML5 files and CSS styling into a single, final executable.
The most important part to think about when getting into JavaScript, is to look closely at the benefits and limitation of each technology.
WebAssembly is fast, wicked fast, and let’s you write code like you are used to from Delphi. Things like pointers etc are supported in Elements, which means ordinary code that use pointers will port over with ease. You are also not bound on hand-and-feet to a particular framework.
For example, EmScripten for C/C++ have almost nothing in terms of UI functionality. The visual part is a custom build of SDL (simple directmedia layer), which fakes the graphics onto an ordinary HTML5 canvas. This makes EmScripten a good candidate for porting games written in C/C++ to the web — but it’s less than optimal for writing serious applications.
Setting up the common tools
So far we have looked at a couple of alternatives for getting into the wonderful world of JavaScript in lieu of other languages. But what if you just want to get started with the typical tools JS developers use?
Visual Studio Code is a pretty amazing code-editor
The first “must have” is Visual Studio Code. This is actually a great example of what you can achieve with JavaScript, because the entire editor and program is written in JavaScript. But I want to stress that this editor is THE editor to get. The way you work with files in JS is very different from Delphi, C# and Java. JavaScript projects are often more fragmented, with less code in each file – organized by name.
TypeScript was invented by Anders Hejlsberg, who also made Delphi and C#
The next “must have” is without a doubt TypeScript. Personally im not too fond of TypeScript, but if ordinary JavaScript makes your head hurt and you want classes and ordinary inheritance, then TypeScript is a step up.
Next on the list is AssemblyScript. This is a post-processor for TypeScript that converts your code into WebAssembly. It lacks much of the charm and elegance of Oxygene, but I suspect that has to do with old habits. When you have been reading object-pascal for 20 years, you feel more at home there.
You will also need to install node.js, which is the runtime engine for running JavaScript as services. Node.js is heavily optimized for writing server software, but it’s actually a brilliant way to write services that are multi-platform. Because Node.js delivers the same behavior regardless of underlying operating system.
And finally, since you definitely want to convert your JavaScript and/or WebAssembly into a stand-alone executable: you will need Adobe Phonegap.
Visual Studio
No matter if you want to enter JavaScript via Elements or something else, Visual Studio will save you a lot of time, especially if you plan on targeting Azure or Amazon services. Downloading and installing the community edition is a good idea, and you can use that while exploring your options.
When it comes to writing system services, you also want to check out NPM, the node.js package manager. The JavaScript ecosystem is heavily package oriented – and npm gives you some 800.000 packages to play with free of charge.
Just to be clear, npm is a shell command you use to install or remove packages. NPM is also a online repository of said packages, where you can search and find what you need. Most packages are hosted on github, but when you install a package locally into your application folder – npm figures out dependencies etc. automatically for you.
Books, glorious books
Last but not least, get some good books. Seriously, it will save you so much time and frustration. Amazon have tons of great books, be it vanilla JavaScript, TypeScript, Node.js — pick some good ones and take the time to consume the material.
And again, I strongly urge you to have a look at Elements when it comes to WebAssembly. WebAssembly is a harsh and barren canvas, and being able to use the Elements RTL is a huge boost.
But regardless of path you pick, you will always benefit from learning vanilla JavaScript.
Generic protect for FPC/Lazarus
Freepascal is not frequently mentioned on my blog. I have written about it from time to time, not always in a positive light though. Just to be clear, FPC (the compiler) is fantastic; it was one particular fork of Lazarus I had issues with, involving a license violation.
On the whole, freepascal and Lazarus is capable of great things. There are a few quirks here and there (if not oddities) that prevents mass adoption (the excessive use of include-files to “fake” partial classes being one), but as object-pascal compilers go, Freepascal is a battle-hardened, production ready system.
It’s been Linux in particular that I have used Freepascal on. In 2015 Hydro Oil wanted to move their back-end from Windows to Linux, and I spent a few months converting windows-only services into Linux daemons.
Today I find myself converting parts of the toolkit I came up with to Oxygene, but that’s a post for another day.
Generic protect
If you work a lot with multithreaded code, the unit im posting here might come in handy. Long story short: sharing composite objects between threads and the main process, always means extra scaffolding. You have to make sure you don’t access the list (or it’s elements) at the same time as another thread for example. To ensure this you can either use a critical-section, or you can deliver the data with a synchronized call. This is more or less universal for all languages, no matter if you are using Oxygene, C/C++, C# or Delphi.
When this unit came into being, I was doing quite elaborate classes with a lot of lists. These classes could not share ancestor, or I could have gotten away with just one locking mechanism. Instead I had to implement the same boilerplate code over and over again.
The unit below makes insulating (or protecting) classes easier. It essentially envelopes whatever class-instance you feed it, and returns the proxy object. Whenever you want to access your instance, you have to unlock it first or use a synchronizer (see below).
Works in both Freepascal and Delphi
The unit works for both Delphi and Freepascal, but there is one little difference. For some reason Freepascal does not support anonymous procedures, so we compensate and use inline-procedures instead. While not a huge deal, I really hope the FPC team add anonymous procedures, it makes life a lot easier for generics based code. Async programming without anonymous procedures is highly impractical too.
So if you are in Delphi you can write:
var lValue: TProtectedValue; lValue.Synchronize( procedure (var Value: integer) begin Value := Value * 12; end);
But under Freepascal you must resort to:
var lValue: TProtectedValue; procedure _UpdateValue(var Data: integer); begin Data := Data * 12; end; begin lValue.Synchronize(@_UpdateValue); end;
On small examples like these, the benefit of this style of coding might be lost; but if you suddenly have 40-50 lists that needs to be shared between 100-200 active threads, it will be a time saver!
You can also use it on intrinsic datatypes:
OK, here we go:
unit safeobjects;
// SafeObjects
// ==========================================================================
// Written by Jon-Lennart Aasenden
// Copyright Quartex Components LTD, all rights reserved
//
// This unit is a part of the QTX Patreon Library
//
// NOTES ABOUT FREEPASCAL:
// =======================
// Freepascal does not allow anonymous procedures, which means we must
// resort to inline procedures instead:
//
// Where we in Delphi could write the following for an atomic,
// thread safe alteration:
//
// var
// LValue: TProtectedValue;
//
// LValue.Synchronize( procedure (var Value: integer)
// begin
// Value := Value * 12;
// end);
//
// Freepascal demands that we use an inline procedure instead, which
// is more or less the same code, just organized slightly differently.
//
// var
// LValue: TProtectedValue;
//
// procedure _UpdateValue(var Data: integer);
// begin
// Data := Data * 12;
// end;
//
// begin
// LValue.Synchronize(@_UpdateValue);
// end;
//
//
//
//
{$mode DELPHI}
{$H+}
interface
uses
{$IFDEF FPC}
SysUtils,
Classes,
SyncObjs,
Generics.Collections;
{$ELSE}
System.SysUtils,
System.Classes,
System.SyncObjs,
System.Generics.Collections;
{$ENDIF}
type
{$DEFINE INHERIT_FROM_CRITICALSECTION}
TProtectedValueAccessRights = set of (lvRead, lvWrite);
EProtectedValue = class(exception);
EProtectedObject = class(exception);
(* Thread safe intrinsic datatype container.
When sharing values between processes, use this class
to make read/write access safe and protected. *)
{$IFDEF INHERIT_FROM_CRITICALSECTION}
TProtectedValue = class(TCriticalSection)
{$ELSE}
TProtectedValue = class(TObject)
{$ENDIF}
strict private
{$IFNDEF INHERIT_FROM_CRITICALSECTION}
FLock: TCriticalSection;
{$ENDIF}
FData: T;
FOptions: TProtectedValueAccessRights;
strict protected
function GetValue: T;virtual;
procedure SetValue(Value: T);virtual;
function GetAccessRights: TProtectedValueAccessRights;
procedure SetAccessRights(Rights: TProtectedValueAccessRights);
public
type
{$IFDEF FPC}
TProtectedValueEntry = procedure (var Data: T);
{$ELSE}
TProtectedValueEntry = reference to procedure (var Data: T);
{$ENDIF}
public
constructor Create(Value: T); overload; virtual;
constructor Create(Value: T; const Access: TProtectedValueAccessRights); overload; virtual;
constructor Create(const Access: TProtectedValueAccessRights); overload; virtual;
destructor Destroy;override;
{$IFNDEF INHERIT_FROM_CRITICALSECTION}
procedure Enter;
procedure Leave;
{$ENDIF}
procedure Synchronize(const Entry: TProtectedValueEntry);
property AccessRights: TProtectedValueAccessRights read GetAccessRights;
property Value: T read GetValue write SetValue;
end;
(* Thread safe object container.
NOTE #1: This object container **CREATES** the instance and maintains it!
Use Edit() to execute a protected block of code with access
to the object.
Note #2: SetValue() does not overwrite the object reference, but
attempts to perform TPersistent.Assign(). If the instance
does not inherit from TPersistent an exception is thrown. *)
TProtectedObject = class(TObject)
strict private
FData: T;
FLock: TCriticalSection;
FOptions: TProtectedValueAccessRights;
strict protected
function GetValue: T;virtual;
procedure SetValue(Value: T);virtual;
function GetAccessRights: TProtectedValueAccessRights;
procedure SetAccessRights(Rights: TProtectedValueAccessRights);
public
type
{$IFDEF FPC}
TProtectedObjectEntry = procedure (const Data: T);
{$ELSE}
TProtectedObjectEntry = reference to procedure (const Data: T);
{$ENDIF}
public
property Value: T read GetValue write SetValue;
property AccessRights: TProtectedValueAccessRights read GetAccessRights;
function Lock: T;
procedure Unlock;
procedure Synchronize(const Entry: TProtectedObjectEntry);
Constructor Create(const AOptions: TProtectedValueAccessRights = [lvRead,lvWrite]); virtual;
Destructor Destroy; override;
end;
(* TProtectedObjectList:
This is a thread-safe object list implementation.
It works more or less like TThreadList, except it deals with objects *)
TProtectedObjectList = class(TInterfacedPersistent)
strict private
FObjects: TObjectList;
FLock: TCriticalSection;
strict protected
function GetEmpty: boolean;virtual;
function GetCount: integer;virtual;
(* QueryObject Proxy: TInterfacedPersistent allows us to
act as a proxy for QueryInterface/GetInterface. Override
and provide another child instance here to expose
interfaces from that instread *)
protected
function GetOwner: TPersistent;override;
public
type
{$IFDEF FPC}
TProtectedObjectListProc = procedure (Item: TObject; var Cancel: boolean);
{$ELSE}
TProtectedObjectListProc = reference to procedure (Item: TObject; var Cancel: boolean);
{$ENDIF}
public
constructor Create(OwnsObjects: Boolean = true); virtual;
destructor Destroy; override;
function Contains(Instance: TObject): boolean; virtual;
function Enter: TObjectList; virtual;
Procedure Leave; virtual;
Procedure Clear; virtual;
procedure ForEach(const CB: TProtectedObjectListProc); virtual;
Property Count: integer read GetCount;
Property Empty: boolean read GetEmpty;
end;
implementation
//############################################################################
// TProtectedObjectList
//############################################################################
constructor TProtectedObjectList.Create(OwnsObjects: Boolean = True);
begin
inherited Create;
FObjects := TObjectList.Create(OwnsObjects);
FLock := TCriticalSection.Create;
end;
destructor TProtectedObjectList.Destroy;
begin
FLock.Enter;
FObjects.Free;
FLock.Free;
inherited;
end;
procedure TProtectedObjectList.Clear;
begin
FLock.Enter;
try
FObjects.Clear;
finally
FLock.Leave;
end;
end;
function TProtectedObjectList.GetOwner: TPersistent;
begin
result := NIL;
end;
procedure TProtectedObjectList.ForEach(const CB: TProtectedObjectListProc);
var
LItem: TObject;
LCancel: Boolean;
begin
LCancel := false;
if assigned(CB) then
begin
FLock.Enter;
try
{$HINTS OFF}
for LItem in FObjects do
begin
LCancel := false;
CB(LItem, LCancel);
if LCancel then
break;
end;
{$HINTS ON}
finally
FLock.Leave;
end;
end;
end;
function TProtectedObjectList.Contains(Instance: TObject): boolean;
begin
result := false;
if assigned(Instance) then
begin
FLock.Enter;
try
result := FObjects.Contains(Instance);
finally
FLock.Leave;
end;
end;
end;
function TProtectedObjectList.GetCount: integer;
begin
FLock.Enter;
try
result :=FObjects.Count;
finally
FLock.Leave;
end;
end;
function TProtectedObjectList.GetEmpty: Boolean;
begin
FLock.Enter;
try
result := FObjects.Count<1;
finally
FLock.Leave;
end;
end;
function TProtectedObjectList.Enter: TObjectList;
begin
FLock.Enter;
result := FObjects;
end;
procedure TProtectedObjectList.Leave;
begin
FLock.Leave;
end;
//############################################################################
// TProtectedObject
//############################################################################
constructor TProtectedObject.Create(const AOptions: TProtectedValueAccessRights = [lvRead, lvWrite]);
begin
inherited Create;
FLock := TCriticalSection.Create;
FLock.Enter();
try
FOptions := AOptions;
FData := T.Create;
finally
FLock.Leave();
end;
end;
destructor TProtectedObject.Destroy;
begin
FData.free;
FLock.Free;
inherited;
end;
function TProtectedObject.GetAccessRights: TProtectedValueAccessRights;
begin
FLock.Enter;
try
result := FOptions;
finally
FLock.Leave;
end;
end;
procedure TProtectedObject.SetAccessRights(Rights: TProtectedValueAccessRights);
begin
FLock.Enter;
try
FOptions := Rights;
finally
FLock.Leave;
end;
end;
function TProtectedObject.Lock: T;
begin
FLock.Enter;
result := FData;
end;
procedure TProtectedObject.Unlock;
begin
FLock.Leave;
end;
procedure TProtectedObject.Synchronize(const Entry: TProtectedObjectEntry);
begin
if assigned(Entry) then
begin
FLock.Enter;
try
Entry(FData);
finally
FLock.Leave;
end;
end;
end;
function TProtectedObject.GetValue: T;
begin
FLock.Enter;
try
if (lvRead in FOptions) then
result := FData
else
raise EProtectedObject.CreateFmt('%s:Read not allowed error',[classname]);
finally
FLock.Leave;
end;
end;
procedure TProtectedObject.SetValue(Value: T);
begin
FLock.Enter;
try
if (lvWrite in FOptions) then
begin
if (TObject(FData) is TPersistent)
or (TObject(FData).InheritsFrom(TPersistent)) then
TPersistent(FData).Assign(TPersistent(Value))
else
raise EProtectedObject.CreateFmt
('Locked object assign failed, %s does not inherit from %s',
[TObject(FData).ClassName,'TPersistent']);
end else
raise EProtectedObject.CreateFmt('%s:Write not allowed error',[classname]);
finally
FLock.Leave;
end;
end;
//############################################################################
// TProtectedValue
//############################################################################
Constructor TProtectedValue.Create(const Access: TProtectedValueAccessRights);
begin
inherited Create;
{$IFNDEF INHERIT_FROM_CRITICALSECTION}
FLock := TCriticalSection.Create;
{$ENDIF}
FOptions := Access;
end;
constructor TProtectedValue.Create(Value: T);
begin
inherited Create;
{$IFNDEF INHERIT_FROM_CRITICALSECTION}
FLock := TCriticalSection.Create;
{$ENDIF}
FOptions := [lvRead, lvWrite];
FData := Value;
end;
constructor TProtectedValue.Create(Value: T; const Access: TProtectedValueAccessRights);
begin
inherited Create;
{$IFNDEF INHERIT_FROM_CRITICALSECTION}
FLock := TCriticalSection.Create;
{$ENDIF}
FOptions := Access;
FData := Value;
end;
Destructor TProtectedValue.Destroy;
begin
{$IFNDEF INHERIT_FROM_CRITICALSECTION}
FLock.Free;
{$ENDIF}
inherited;
end;
function TProtectedValue.GetAccessRights: TProtectedValueAccessRights;
begin
Enter();
try
result := FOptions;
finally
Leave();
end;
end;
procedure TProtectedValue.SetAccessRights(Rights: TProtectedValueAccessRights);
begin
Enter();
try
FOptions := Rights;
finally
Leave();
end;
end;
{$IFNDEF INHERIT_FROM_CRITICALSECTION}
procedure TProtectedValue.Enter;
begin
FLock.Enter;
end;
procedure TProtectedValue.Leave;
begin
FLock.Leave;
end;
{$ENDIF}
procedure TProtectedValue.Synchronize(const Entry: TProtectedValueEntry);
begin
if assigned(Entry) then
Begin
Enter();
try
Entry(FData);
finally
Leave();
end;
end;
end;
function TProtectedValue.GetValue: T;
begin
Enter();
try
if (lvRead in FOptions) then
result := FData
else
raise EProtectedValue.CreateFmt('%s: Read not allowed error', [Classname]);
finally
Leave();
end;
end;
procedure TProtectedValue.SetValue(Value: T);
begin
Enter();
try
if (lvWrite in FOptions) then
FData:=Value
else
raise EProtectedValue.CreateFmt('%s: Write not allowed error', [Classname]);
finally
Leave();
end;
end;
end.
Raspberry PI 4 at last!
It was with astonishment that I opened up my browser this morning to read some daily IT news, only to discover that the Raspberry PI v4 has finally arrived! And boy what a landslide update to the 3.x family it is!
Three times the fun
There are plenty of sites that entertains page-up and page-down with numbers, but I will save all that for an article where I have the physical kit in my posession. But looking at the preliminaries I think it’s safe to say that we are looking at a solid 3x the speed of the older yet capable PI 3b+.
The PI returns, and what a joy it is!
While the 3x speed boost is enough to bump the SoC up, from entertaining to serious for business applications – it’s ultimately the memory footprint that will make all the difference. While the Raspberry PI is probably the most loved SBC (single board computer) of all time, it’s always been cut short due to lack of memory. 512 megabyte can only do so much in 2019, and even the slimmest of Linux distributions quickly consumes more ram that older versions could supply.
VideoCore 6, two screens and 4k video
The new model ships in three different configurations, with 1, 2 and 4 gigabytes of ram respectively. I strongly urge people to get the 4Gb version, because with that amount of memory coupled with a good solid-state-disk, means you can enable a proper swap-partition. No matter how fast a SoC might be, without memory to compliment it – the system simply wont be able to deliver on its potential. But with 4Gb, a nice solid state disk (just use a SSD-To-USB with one of the sexy new USB 3.x ports) and you are looking at an OK mini-computer capable of most desktop applications.
I have to admit I never expected the PI to ship with support for two monitors, but lo-and-behold, the board has two mini-hdmi out ports! The board is also fitted with the VideCore 6 rather than VideoCore 4.
Not missing the boat with Oxygene and Elements
One of the most frustrating episodes in the history of Delphi, is that we never got a Delphi edition that could target Raspberry PI (or ARM-Linux in general). It was especially frustrating since Allen Bauer actually demonstrated Delphi generating code that ran on a PI in 2012. The result of not directly supporting the PI, even on service level without a UI layer – is that Delphi developers have missed the IOT market completely.
Before Delphi developers missed the IOT revolution, Delphi also missed out on iOS and Android. By the time Delphi developers could target any of these platforms, the market was completely saturated, and all opportunities to make money was long gone. In other words, Delphi has missed the boat on 3 revolutionary platforms in a row. Something which is borderline unforgivable.
The good news though is that Oxygene, the object-pascal compiler from RemObjects, supports the Raspberry PI SoC. I have yet to test this on v4, but since the PI v4 is 100% backwards compatible I don’t see any reason why there should be any issues. The code generated by Oxygene is not bound to just the PI either. As long as it runs on a debian based distro, it should run just fine on most ARM-Linux SoC’s that have working drivers.
And like I have written about elsewhere, you can also compile for WebAssembly, running either in node.js or in the browser — so there are plenty of ways to get your products over!
Stay tuned for the lineup
This week im going to do a lot of testing on various ARM devices to find out just how many SBC’s Oxygene can target, starting with the ODroid N2. But for Raspberry PI, that should be a slam-dunk. Meaning that object-pascal developers can finally make use of affordable off-the-shelves parts in their hardware projects.
As of writing im preparing the various boards I will be testing. We have the PI 3b+, the Tinkerboard from ASUS, NanoPI, Dragonboard, Odroid XU4 – and the latest power-board, the ODroid N2. Out of these offerings only the N2 is en-par with the Raspberry PI v4, although I suspect the Videocore 6 GPU will outperform the Mali G52.
Hydra now supports Freepascal and Java
In case you guys missed it, RemObjects Hydra 6.2 now supports FreePascal!
This means that you can now use forms and units from .net and Java from your Freepascal applications – and (drumroll) also mix and match between Delphi, .net, Java and FPC modules! So if you see something cool that Freepascal lacks, just slap it in a Hydra module and you can use it across language barriers.
I have used Hydra for years with Delphi, and being able to use .net forms and components in Delphi is pretty awesome. It’s also a great framework for building modular applications that are easier to manage.
Being able to tap into Freepascal is a great feature. Or the other way around, with Freepascal showing forms from Delphi, .net or Java.
For example, if you are moving to Freepascal, you can isolate the forms or controls that are not available under Freepascal in a Hydra module, and voila – you can gradually migrate.
If you are moving to Oxygene Pascal the same applies, you can implement the immediate logic under .net, and then import and use the parts that can’t easily be ported (or that you want to wait with).
The best of four worlds — You gotta love that!
Check out Hydra here:
RemObjects Remoting SDK?
Reading this you could be forgiven for thinking that I must promote RemObjects products, It’s my job now right? Well yes, but also no.
The thing is, I’m really not “traveling salesman” material by any stretch of the imagination. My tolerance for bullshit is ridiculously low, and being practical of nature I loath fancy products that cost a fortune yet deliver nothing but superficial fluff.
The reasons I went to work at RemObjects are many, but most of all it’s because I have been an avid supporter of their products since they launched. I have used and seen their products in action under intense pressure, and I have come to put some faith in their solutions.
Trying to describe what it’s like to write servers that should handle thousands of active user “with or without” RemObjects Remoting SDK is exhausting, because you end up sounding like a fanatic. Having said that, I feel comfortable talking about the products because I speak from experience.
I will try to outline some of the benefits here, but you really should check it out yourself. You can download a trial directly here: https://www.remotingsdk.com/ro/
Remoting framework, what’s that?
RemObjects Remoting framework (or “RemObjects SDK” as it was called earlier) is a framework for writing large-scale RPC (remote procedure call) servers and services. Unlike the typical solutions available for Delphi and C++ builder, including those from Embarcadero I might add, RemObjects framework stands out because it distinguishes between transport, host and message-format – and above all, it’s sheer quality and ease of use.
RemObjects Remoting SDK ships with a rich selection of channels and message formats
This separation between transport, host and message-format makes a lot of sense, because the parameters and data involved in calling a server-method, shouldn’t really be affected by how it got there.
And this is where the fun begins because the framework offers you a great deal of different server types (channels) and you can put together some interesting combinations by just dragging and dropping components.
How about JSON over email? Or XML over pipes?
The whole idea here is that you don’t have to just work with one standard (and pay through the nose for the privilege). You can mix and match from a rich palette of transport mediums and message-formats and instead focus on your job; to deliver a kick-ass product.
And should you need something special that isn’t covered by the existing components, inheriting out your own channel or message classes is likewise a breeze. For example, Andre Mussche have some additional components on GitHub that adds a WebSocket server and client. So there is a lot of room for expanding and building on the foundation provided by RemObjects.
And this is where RemObjects has the biggest edge (imho), namely that their solutions shaves weeks if not months off your development time. And the central aspect of that is their integrated service designer.
Integration into the Delphi IDE
Dropping components on a form is all good and well, but the moment you start coding services that deploy complex data-types (records or structures) the amount of boilerplate code can become overwhelming.
The whole point of a remoting framework is that it should expose your services to the world. Someone working in .net or Java on the other side of the planet should be able to connect, consume and invoke your services. And for that to happen every minute detail of your service has to follow standards.
The RemObjects Service Builder integrates directly into the Delphi IDE
When you install RemObjects SDK, it also integrates into the Delphi IDE. And one of the features it integrates is a complete, separate service designer. The designer can also be used outside of the Delphi IDE, but I cannot underline enough how handy it is to be able to design your services visually, right there and then, in the Delphi IDE.
This designer doesn’t just help you design your service description (RemObjects has their own RODL file-format, which is a bit like a Microsoft WSDL file), the core purpose is to auto-generate all the boilerplate code for you — directly into your Delphi project (!)
So instead of you having to spend a week typing boilerplate code for your killer solution, you get to focus on implementing the actual methods (which is what you are supposed to be doing in the first place).
DLL services, code re-use and multi-tenancy
The idea of multi-tenancy is an interesting one. One that I talked about with regards to Rad-Server both in Oslo and London before christmas. But Rad-Server is not the only system that allows for multi-tenancy. I was doing multi-tenancy with RemObjects SDK some 14 years ago (if not earlier).
Remember how I said the framework distinguishes between transport, message and host? That last bit, namely host, is going to change how you write applications.
When you install the framework, it registers a series of custom project types inside the Delphi IDE. So if you want to create a brand new RemObjects SDK server project, you can just do that via the ordinary File->New->Other menu option.
One of the project types is called a DLL Server. Which literally means you get to isolate a whole service library inside a single DLL file! You can then load in this DLL file and call the functions from other projects. And that is, ultimately, the fundamental principle for multi-tenancy.
And no, you don’t have to compile your project with external packages for this to work. The term “dll-server” can also be a bit confusing, because we are not compiling a network server into a DLL file, we are placing the code for a service into a DLL file. I used this project type to isolate common code, so I wouldn’t have to copy unit-files all over the place when delivering the same functionality.
It’s also a great way to save money. Don’t want to pay for that new upgrade? Happy with the database components you have? Isolate them in a DLL-Server and continue to use the code from your new Delphi edition. I have Delphi XE3 Database components running inside a RemObjects DLL-Server that I use from Delphi XE 10.3.
DLL server is awesome and elegantly solves real-life problems out of the box
In my example I was doing business-logic for our biggest customers. Each of them used the same database, but they way they registered data was different. The company I worked for had bought up these projects (and thus their customers with them), and in order to keep the customers happy we couldn’t force them to re-code their systems to match ours. So we had to come up with a way to upgrade our technology without forcing a change on them.
The first thing I did was to create a “DLL server” that dealt with the database. It exposed methods like openTable(), createInvoice(), getInvoiceById() and so on. All the functions I would need to work with the data without getting my fingers dirty with SQL outside the DLL. So all the nitty gritty of SQL components, queries and whatnot was neatly isolated in that DLL file.
I then created separate DLL-Server projects for each customer, implemented their service interfaces identical to their older API. These DLL’s directly referenced the database library for authentication and doing the actual work.
When integrated with the IDE, you are greeted with a nice welcome window when you start Delphi. Here you can open examples or check out the documentation
Finally, I wrapped it all up in a traditional Windows system service, which contained two different server-channels and the message formats they needed. When the service was started it would simply load in the DLL’s and manually register their services and types with the central channel — and voila, it worked like a charm!
Rock solid
Some 10 years after I delivered the RemObjects based solution outlined above, I got a call from my old employer. They had been victim of a devastating cyber attack. I got a bit anxious as he went on and on about damages and costs, fearing that I had somehow contributed to the situation.
But it turned out he called to congratulate me! Out of all the services in their server-park, mine were the only ones left standing when the dust settled.
The RemObjects payload balancer had correctly dealt with both DDOS and brute force attacks, and the hackers were left wanting at the gates.
New job, new office, new adventures
It’s been roughly 4 weeks since I posted a status report on Amibian.js. I normally keep people up-to-date on facebook (the “Amiga Disrupt” and also “Delphi Developer” groups). It’s been a very hectic month so I fully understand that people are asking. So let’s look at where the project is at and where we are on the time-line.
For those that might not know, I decided to leave Embarcadero a couple of months ago. I will be working out may before I move on. I wanted to write about that myself in a clean fashion, but sadly the news broke on Facebook prematurely.
Long story short, I have been very fortunate to work at Embarcadero. I am not leaving because there is anything wrong or something like that. I was hired as SC for the EMEA regions, which basically made me the support and presenter for most of europe, parts of asia and the middle east. It’s been a great adventure, but ultimately I had to admit that my passion is coding and community work. Sales is a very important part of any company, but it’s not really my cup of tea; my passion has always been research and development.
So, come first of June and I start in a new position at RemObjects. A company that has deep roots with Delphi and C++ builder users – and a company that continues to produce a wealth of high-quality, high-performance frameworks for Delphi and C++ builder. RemObjects also has a strong focus on modern languages, and have a strong portfolio of new and exciting compilers and languages to offer. The Oxygene compiler should be no stranger to Delphi developers, a powerful object-pascal dialect that can target a variety of platforms and chipsets.
Since compiler technology and run-time systems has been my main focus for well over a decade now, I feel RemObjects is a better match.
Quartex Components
Quartex Components has been an officially registered Norwegian company for a while now, so perhaps not news. What is news is that it’s now directly connected with the development of the Quartex Media Desktop (codename “Amibian.js”). While Amibian.js is an open source endeavour, there will be both free and commercial products running on top of that platform. I have written at length about Cloud Forge in the past, so I wont re-hash that again. But 2020 will see a paradigm shift in how teams and companies approach software development.
Company logo professionally milled and on its way to my new office
I will also, once there is more time, continue to sell and support software license components.
Quartex Media Desktop
The “Amibian.js” project is moving along nicely. The deadline is Q4 2019, but im hoping to wrap up the core functionality before that. So we are on track and kicking ass 🙂
More and more elaborate functionality is being implemented for the desktop
Here is an overview of work done this month:
- TSystemService application type has been created (node.js)
- TApplication now holds IPC functions (inter process communication)
- Running child processes + sending messages is now simplicity itself
- Database drivers are 90% done. Delete() and DeleteTable() functionality needs to be implemented in a uniform way
- Authentication is now a separate service
- Service database layer is finished (using SQLite3 driver by default)
- Authentication protocol has been designed
- Server protocol and JSON message envelopes are done
- Presently working on the client interface
- LDEF bytecode assembler has been improved
- Faster symbolic lookup
- Smarter register recognition
- Early support for stack-frames
- Fixed bug in parser (comma-list parse)
- QTX framework has seen a lot of work
- Large parts of the RTL sub-strata has been implemented
- UTF16 codec implemented
- QTX versions of common controls:
- TQTXButton
- TQTXLabel
- TQTXToolbar
- TQTXToolButton
- TQTXToolSeparator
- TQTXToolElement
- TQTXPanel
- TQTXCheckBox
- .. and much, much more
- Desktop changes
- Link Maker functionality has been added
- Handshake process between desktop and child app now runs on a separate timer, ensuring better conformity and a more robust initialization
- The Quartex Editor control has been optimized
- All redraw calls are now synchronized
- Canvas is created on demand, avoids flicker during initial redraw
- Support for DEL key + behavior
- Gutter is now rendered to an offscreen bitmap and blitted into the control’s canvas. The gutter is only fully rendered when cursor forces the view to change
I will continue to keep everyone up to date about the project. As you can understand, its a bit hectic right now so please be patient – it is turning into an EPIC environment!
Understanding a stack
The concept of stacks is an old one, and together with linked-lists and queues – these form the most fundamental programming concepts a developer needs to master.
But, the stack most people use today in languages like object pascal and C++ are not actual stacks; they are more like “conveniently repurposed lists“. Not a huge issue I agree, but the misconception is enough to cause confusion when people dive into low-level programming.
Adventures in assembly-land
It might seem odd to focus on something as trivial as a stack, but I have my reasons. A friend of mine who is a brilliant coder with plenty of large projects behind him recently decided to have a go at assembly coding. He was doing fine and everything was great, until he started pushing and popping things off the stack.
After a little chat I realized that the problem was not his code, but rather how he viewed the stack. He was used to high-level versions of stacks, which in most cases are just lists storing arbitrary sized data – so he was looking at the stack as a TList<item> expecting similar behavior. Superficially a real-stack and a list-stack work the same if all you do is clean push and pop operations, but the moment you start designing a stack-scheme and push more elaborate constructs (stack-frames), things can go wrong really fast.
The nature of a real stack
A “real” stack that is a part of a hardware SOC (system on a chip) has nothing to do with lists. It’s actually a solid chunk of memory with a register to keep track of an offset into this memory block.
Let’s for sake of argument say you have 4k of stack space right? It’s clean and contains nothing, so the SP (stack pointer, or offset) is zero. What happens when you push something to the stack? for example:
push EDX
The code above simply writes the content of the EDX register to whatever offset the SP contains. It then updates the SP with the size of the data (EDX is a 32bit register, so the SP is incremented by a longword or 4 bytes). In Delphi pseudocode what happens is something like:
var LAddr := FStackBuffer; inc(LAddr, SP); PLongword(LAddr)^ := EDX; inc(SP, SizeOf(EDX));
The thing about a stack is that it doesn’t manage data-length for you. And that is a big difference to remember. It will push or pop data based on the size of the source (in this case the 32bit EDX register) you use.
If you push 1024 bytes of data to a list based stack, the list keeps track of the size and data for you. So when you pop the data from the stack, you get back that data regardless. But a “real” stack couldn’t care less — which is also why it’s so easy to screw up an entire program if you make a mistake.
In short: The length of what you push – must be matched when you pop the data back (!) If you push a longword, you MUST pop a longword later.
Benefits of a real stack
The benefit is that the cost of storing values on a stack is almost zero in terms of cpu operations. A list based stack is more expensive; it will allocate memory for a record-item to hold the information about the data, then it will allocate memory to hold the actual data (depends on the type naturally) and finally copy the data into the newly allocated buffer. Hundreds if not thousands of instructions can be involved here.
A real stack will just write whatever you pushed directly into the stack-memory at whatever offset SP is at. Once written it will add the length of the write to the SP – and that’s it! So it’s one of the oldest and fastest mechanisms for lining up data in a predictable way.
Again the rules are simple: when you pop something off the stack, the size must match whatever you used to push it there. So if you pushed a longword (EDX) you also have to make sure you use a 32-bit target when you pop the value back. If you use RDX, which is 64 bit then you will essentially steal 4 bytes from something else using that stack – and all hell will break loose down the line.
Stack schemes and frames
Im not going to dig too deeply into stack-frames here, but instead write a few words about stack-schemes and using the stack to persist data your code relies on. The lines blur between those two topics anyways.
The SP (stack pointer) is not just a simple offset you can read, you can also write and change it (it also serves as a pointer). You can also read from whatever memory the SP is pointing at without polling any data from the stack.
What language developers usually do, is that they design entire structures on the stack that are, when you get into the nitty-gritty, “offset based records”. For example, lets say you have a record that looks like this:
type PMyRecord ) ^TMyRecord; TMyRecord = record first: Pointer; second: integer; Third: array[0..255] of longword; end;
Instead of allocating conventional ram to hold that record, people push it to the stack and then use offsets to read and update the values there. A bit like a super global variable if you like. This is why when you disassemble code, you find stuff like:
mov EDX, (SP)+4
If the above record was on the stack, that pseudo code would move the field “second” into the EDX register. Because that field is 4 bytes from the stack start (providing SP points to zero).
Every programming language has a stack scheme to keep track of things. Local variables, global variables, class instances, type RTTI — most of these things are allocated in conventional ram – but there is a “program record” on the stack that makes it easy to access that information quickly.
This “moving a whole record onto the stack” is basically what a stack-frame is all about. It used to be a very costly affair with a heavy cpu speed penalty. If you look in your Delphi compiler options you will see that there is a checkbox regarding this very topic. Delphi can be told to avoid stack-frames and do register allocation instead, which was super quick compared to stack-frames – but CPU’s today are largely optimized for stack-frame allocation as default, so I doubt there is much to gain by this in 2019.
Note: A stack frame is much more, but its out of scope for this post. Google it for more info.
To sum up
When doing high-level coding you don’t really need to bother with the nuances between a TStack<item> and a “real” stack. But if you plan on digging deeper and learning a few lines of assembly – learning the differences is imperative. Its boring stuff but as fundamental as wheels on a bicycle. There is no way to avoid it, so might as well jump in.
In its absolute raw form, here is roughly the same functionality for Delphi. This was written on the fly in 2 minutes while on the road, so its purely to give you a rough idea of behavior. I would add a secondary field to keep track of the end (next insertion point), that way SP can be changed without overwriting data on new pushes.
And yes, wrapping this in a TObject utterly defeats the purpose of low-level coding, but hopefully it gives you some idea of the differences 🙂
Delphi AST, XML and weekend experiments
One of the benefits of the Delphi IDE is that it’s a very rich eco-system that component writers and technology partners can tap into for their own products. I know that writing your own components is not something everyone enjoy, but knowing that you can in-fact write tools that expands the IDE using just Delphi or C++ builder, opens up for some interesting tools.
Ye old compiler bible
Delphi has a long tradition of “IDE enhancement” software and elaborate third-party tools that automate or delivers some benefit right in the environment. RemObjects SDK is probably the best example of how flexible the IDE truly is. RemObjects SDK integrates a whole service designer, which will generate source-code for you, update the code if you change something – and even generate service manifests for you.
There are also other tools that show off the flexibility of the IDE, ranging from code migration to advanced code refactoring and optimization.
It was with the last bit, namely code refactoring, that a third-party open-source library received a lot of deserving attention a couple of years back. A package called DelphiAST. This is a low-level syntax parser that reads Delphi source-code, applies fundamental syntax checks, and transforms the code into XML. A wet dream for anyone interested in writing advanced tooling that operates directly on source-code level.
Delphi AST
Like mentioned above, DelphiAST is a parser. Its job is very simple: parse the code, perform language level syntax checking, and convert each aspect of the code to a valid XML element. We are not talking about stuffing source-code into a CDATA segment here, but rather breaking each statement into separate tags (begin, end, if, procedure, param) so you can apply filtering, transformations and everything XML has to offer.
Back when Roman first started on DelphiAST, I got thinking — could we follow this idea further, and apply XML transformation to produce something more interesting? Would it actually be possible to approach the notion of compiling from a whole new angle? Perhaps convert between languages in a more effective way?
The short answer is: yes, everything is possible. But as always there are caveats and obstacles to overcome.
First of all, DelphiAST despite its name doesn’t actually generate a fully functional abstract symbol tree (AST). It generates a data model that is very suitable for AST generation, but not an actual AST. Everything in a programming language that can be referenced, like a method, a class, a global variable, a local variable, a parameter – are all called “symbols”. And before you can even think about processing the code, a fast and reliable AST must be in place.
Who cares?
Before I continue, you might be wondering why re-inventing the wheel is even a thing here? Why would anyone research compilers in 2019 when the world is abundant with compilers for a multitude of languages?
Because the world of computing is about to be hit by a tsunami, that’s why.
Quartex Pascal
In the next 8-10 years the world of computing will be turned on its head. NVIDIA and roughly 100 tech companies have invested in open-source CPU designs, making it very clear that playing by Intel’s rules and bleeding royalties will no longer be tolerated. IBM has woken up from its “patent induced slumber” and is set to push their P9 cpu architecture, targeting both the high-end server and embedded market (see my article last year on PPC). At the same time Microsoft and Apple have both signaled that they are moving to ARM (an estimate of 5 years is probably reasonable). Laptop beta’s are said to be already rolling, with a commercial version expected Q3 this year (I think it wont arrive before xmas, but who knows).
Intel has remained somewhat silent about any long-term plans, but everyone that keeps an eye on hardware knows they are working like mad on next-gen FPGA. A tech that has the potential to disrupt the whole industry. Work is also being done to bridge FPGA coding with traditional code; there is no way of predicting the outcome of that though.
Oh and AMD is usurping the Intel marketshare at a steady rate — so we are in for a fight to the death.
The rise of C/C++
Those that keep tabs on languages have no doubt noticed the spike in C/C++ popularity lately. And the cause of this is that developers are safeguarding themselves for the storm to come. C as a language might not be the most beautiful out there, but truth be told, it’s tooling requires the least amount of work to target a new platform. When a new architecture is released, C/C++ is always the first language available. You wont see C#, Flutter or Rust shipping with the latest and greatest; It’s always GCC or Clang.
Note: GCC is not just C, it’s actually a family of languages, so ironically, Gnu Basic hits a platform at the same time.
Those that have followed my blog for the past 10 years, should be more than aware of my experiments. From compiling to Javascript, generating bytecodes – and right now, moving the whole development paradigm to the browser. Hopefully my readers also recognize why this is important.
But to make you understand why I am so passionate about my compiler experiments, let’s do a little thought experiment:
Rethinking tooling
Let’s say we take Delphi, implement a bytecode format and streamline the RTL to be platform agnostic. What would the consequences of that be?
Well, first of all the compiler process would be split in two. The traditional compilation process would still be there, but it would generate bytecodes rather than machine code. That part would be isolated in a completely separate process; a process that, just like with the Delphi IDE’s infrastructure, could be outsourced to component-writers and technology partners. This in turn would provide the community with a high degree of safety, since the community itself could approach new targets without waiting for Embarcadero.
Even more, such an architecture would not be limited to machine-code. There is no law that says “you must convert bytecodes to machine code”. Since C/C++ is the foundation that modern operating-systems rest on, generating C/C++ source-code that can be built by existing compilers is a valid strategy.
There is also another factor to include in all of this, and that is Linux. Borland was correct in their assessment of Linux (the Kylix project), but they failed miserably with regards to timing. They also gravely underestimated Linux user’s sense of quality, depending on Wine (a Windows virtualization framework) to even function. They also underestimated Freepascal and Lazarus, because Linux is something FPC does exceptionally well. Competing financially against free products wont work unless you bring outstanding abilities to the table. And Linux have development tools that rival Visual Studio in quality, yet costs nothing.
But no matter how financially tricky Linux might be, we have reached the point in time where Linux is becoming mainstream. 10 years ago I had to setup my own Linux machine. There were no retailers locally that shipped a Linux box. Today I can walk into two major chains and pick dedicated Linux machines. Ubuntu in particular is well established and delivers LTS.
So for me personally, compiler tech has never been more important. And even more important is the tooling being universal and unbound by any specific API or cpu instruction-set. Firemonkey is absolutely a step in the right direction, but I think it’s a disaster to focus on native UI’s beyond a system level binding. Because replicating the same level of support and functionality for ARM, P9, RISC 5 and whatever monstrosity Intel comes up with through FPGA will take forever.
Transformation based conversion
We have wandered far off topic now, so let’s bring it back to this weekends experiment.
In short, XML transformations to convert code does work, but the right tooling have to be there to make it viable. I implemented a poor-man’s symbol table, just collecting classes, types and methods – and yeah, works just fine. What worries me a bit though is the XML parser. Microsoft has put a lot of money into XML file handling on enterprise level. When working with massive XML files (read: gigabytes) you really can’t be bothered to load the file into conventional ram and then old-school traverse the XML character by character. Microsoft operates with pure memory mapping so that you can process gigabytes like they were megabytes — but sadly, there is nothing similar for Linux, Unix or Android, that abruptly ends the fascination for me.
The only place I see using XML transformations to process source-code, is when converting to another language on source-level.
So the idea, although technically sound, gives zero benefits over the traditional process. I am however very interested in using DelphiAST to analyze and convert Delphi code directly from the IDE. But that will have to be an experiment for 2020, im booked 24/7 with Quartex Media Desktop right now.
But it was great fun playing around with DelphiAST! I loved how clean and neat the codebase has become. So if you need to work with source-code, DelphiAST is just the ticket!
Edit: You dont have to emit the code as XML. DelphiAST is perfectly happy to act as a clean parser, just saying.
TTween library for Delphi now free
I have asked for financial backing while creating libraries that people want and enjoy, and as promised they are released into open-source land afterwards.
HexLicense was open-sourced a while back, and this time it’s TTween library that is going back to the community.
Tweening?
You have probably noticed how mobile phone UI’s have smooth movements? like on iOS when you click “back” the whole display slides smoothly into view; or that elements move, grow and shrink using fancy, accelerated effects?
This type of animation is called tweening. And the TTween Library makes it super easy to do the same for your VCL applications.
Check out this Youtube video to see how you can make your VCL apps scale their controls more smoothly
You can fork the project here: https://bitbucket.org/cipher_diaz/ttween/src/master/
To install the system as ordinary components, just open the “Tweening.dproj” file and install as normal. Remember to add the directory to your libraries path!
Support the cause
If you like my articles and want to see more libraries and techniques, then consider donating to the project here: https://www.paypal.me/quartexNOR
Those that donate $50 or more automatically get access to the Quartex Web OS repositories, including full access to the QTX replacement RTL (for DWScript and Smart Mobile Studio).
Thank you for your support, projects like Amibian.js and the Quartex Web OS would not exist without my backers!
Building a Delphi Database engine, part four
This article is over six months late (gasp!). Work at Embarcadero have been extremely time consuming, and my free time has been bound up in my ex-patreon project. So that’s why I was unable to finish in a more predictable fashion.
But better late than never — and we have finally reached one of the more exciting steps in the evolution of our database engine design, namely the place where we link our metadata to actual data.
So far we have been busy with the underlying mechanisms, how to split up larger pieces of data, how to collect these pieces and re-assemble them, how to grow and scale the database file and so on.
We ended our last article with a working persistence layer, meaning that the codebase is now able to write the metadata to itself, read it back when you open the database, persist sequences (records) – and our humble API is now rich enough to handle tasks like scaling. At the present we only support growth, but we can add file compacting later.
Tables and records
In our last article’s code, the metadata exposed a Table class. This table-class in turn exposed an interface to our field-definitions, so that we have a way to define how a table should look before we create the database.
You have probably taken a look at the code (I hope so, or much of this won’t make much sense) and noticed that the record class (TDbLibRecord) is used both as a blueprint for a table (field definitions), as well as the actual class that holds the values.
If you look at the class again (TDbLibRecord can be found in the file dblib.records.pas), you will notice that it has a series of interfaces attached to it:
- IDbLibFields
- IStreamPersist
The first one, which we expose in our Table as the FieldDefs property, simply exposes functions for adding and working with the fields. While somewhat different from Delphi’s traditional TFieldDefinition class, it’s familiar enough. I don’t think anyone who has used Delphi with databases would be confused around it’s members:
IDbLibFields = interface
['{0D6A9FE2-24D2-42AE-A343-E65F18409FA2}']
function IndexOf(FieldName: string): integer;
function ObjectOf(FieldName: string): TDbLibRecordField;
function Add(const FieldName: string; const FieldClass: TDbLibRecordFieldClass): TDbLibRecordField;
function Addinteger(const FieldName: string): TDbLibFieldInteger;
function AddStr(const FieldName: string): TDbLibFieldString;
function Addbyte(const FieldName: string): TDbLibFieldbyte;
function AddBool(const FieldName: string): TDbLibFieldboolean;
function AddCurrency(const FieldName: string): TDbLibFieldCurrency;
function AddData(const FieldName: string): TDbLibFieldData;
function AddDateTime(const FieldName: string): TDbLibFieldDateTime;
function AddDouble(const FieldName: string): TDbLibFieldDouble;
function AddGUID(const FieldName: string): TDbLibFieldGUID;
function AddInt64(const FieldName: string): TDbLibFieldInt64;
function AddLong(const FieldName: string): TDbLibFieldLong;
end;
But, as you can see, this interface is just a small part of what the class is actually about. The class can indeed hold a list of fields, each with its own datatype – but it can also persist these fields to a stream and read them back again. You can also read and write a value to each field. So it is, for all means and purposes, a single record in class form.
The term people use for this type of class is: property bag, and it was a part of the Microsoft standard components (Active X / COM) for ages. Its probably still there, but I prefer my own take on the system.
In this article we are going to finish that work, namely the ability to define a table, create a database based on the metadata, insert a new record, read records, and push the resulting binary data to the database file. And since the persistency is already in place, opening the database and reading the record back is pretty straight forward.
So this is where the metadata stops being just a blue-print, and becomes something tangible and real.
Who owns what?
Before we continue, we have to stop and think about ownership. Right now the database file persists a global list of sequences. The database class itself has no interest in who owns each sequence, if a sequence belongs to a table, if it contains a picture, a number or whatever the content might be — it simply keeps track of where each sequence begins.
So the first order of the day is to expand the metadata for tables to manage whatever records belongs to that table. In short, the database class will focus on data within its scope, and the table instances will maintain their own overview.
So the metadata suddenly need to save a list of longwords with each table. You might say that this is wasteful, that the list maintained by the database should be eliminated and that each table should keep track of it’s own data. And while that is tempting to do, there is also something to be said about maintenance. Being able to deal with persisted data without getting involved with the nitty-gritty of tables is going to be useful when things like database compacting enters at the end of our tutorial.
Locking mechanism
Delphi has a very user-friendly locking mechanism when it comes to databases. A table or dataset is either in read, edit or insert mode – and various functions are allowed or prohibited depending on that state. And it would probably be wise to merge the engine with Delphi’s own TDatabase and TTable at some point – but right now im more interested in keeping things clean and simple.
When I write “locking mechanism” I am not referring to a file-lock, or memory lock. Had we used memory-mapped files the locking mechanism would have been more elaborate. What I mean with a lock, is basically placing a table in one of the states I mentioned above. The table needs to know what exactly you want to do. Are you adding a record? Are you editing an existing record? The table code needs to know this to safely bring you from one mode to the next.
Suddenly, you realize why each table needs that extra list, because how is the table going to allow methods like first, next, last and previous? The record-list dealt with by the database is just a generic, non-ordered ledger of sequences (a global scope list if you will). Are you going to read all records back when you open the database to figure out who owns what?
A call to First() will mean a completely different offset for each table. And the logical way to handle this, is to give each table it’s own cursor. A class that keeps track of what records belongs to the table, and also keeps track of whatever states the table is in.
The database cursor
Since we are not up against Oracle or MSSQL here, but exploring database theory, I have kept the cursor as simple as I possibly could. It is a humble class that looks like this:
The idea of-course is that the table defaults to “read” mode, meaning that you can navigate around, record by record, or jump to a specific record using the traditional RecNo property.
The moment you want to insert or edit a record, you call the Lock() method, passing along the locking you need (edit or insert). You can then either cancel the operation or call post() to push the data down to the file.
The Lock() method is a function (bool), making it easier to write code, as such:
if Database.Table.Cursor.Lock(cmInsert) then
begin
with Database.GetTableByName('access_log').cursor do
begin
Fields.WriteInt('id', FUserId);
Fields.WriteStr('name', FuserName);
Fields.WriteDateTime('access', Now);
Post();
end;
end else
raise exception.create('failed to insert record');
Im sure the are better designs, and the classes and layout can absolutely be made better; but for our purposes it should be more than adequate.
Reloading record data
In the previous articles we focused on writing data. Basically taking a stream or a buffer, breaking it into pages, and then storing the pages (or blocks) around the file where there was available space.
We cleverly crafted the blocks so that they would contain the offset to the next block in a sequence, making it possible to read back a whole sequence of blocks by just knowing the first one (!)
A part of what the cursor does is also to read data back. Whenever the RecNo field changes, meaning that you are moving around the table-records using the typical Next(), Previous(), First() etc functions — if the cursor is in read mode (meaning: you are not inserting data, nor are you editing an existing record), you have to read the record into memory. Otherwise the in-memory fields wont contain the data for that record.
Creating a cursor
One note before you dive into the code: You have to create a cursor before you can use it! So just creating a table etc wont be enough. Here is how you go about doing this:
Creating the cursor will be neatly tucked into a function for the table instance, we still have other issues to deal with.
What to expect next?
Next time we will be looking at editing a record, commiting changes and deleting records. And with that in place we have finally reached the point where we can add more elaborate functionality, starting with expression parsing and filters!
You can check out the code here: https://bitbucket.org/cipher_diaz/dbproject/src/master/
Support the cause
If you like my articles and want to see more libraries and techniques, then consider donating to the project here: https://www.paypal.me/quartexNOR
Those that donate $50 or more automatically get access to the Quartex Web OS repositories, including full access to the QTX replacement RTL (for DWScript and Smart Mobile Studio).
Thank you for your support, projects like Amibian.js and the Quartex Web OS would not exist without my backers!
/Jon
VMWare: A Delphi developers best friend
Full disclosure: I am not affiliated with any particular virtualization vendor of any sorts. The reason I picked VMWare was because their product was faster when I compared the various solutions. So feel free to replace the word VMWare with whatever virtualization software suits your needs.
On Delphi Developer we get new members and questions about Delphi and C++ builder every day. It’s grown into an awesome community where we help each other, do business, find jobs and even become personal friends.
A part of what we do in our community, is to tip each other about cool stuff. It doesn’t have to be directly bound to Delphi or code either; people have posted open source graphic programs, video editing, database designers – as long as its open source or freeware its a great thing (we have a strict policy of no piracy or illegal copying).
Today we got talking about VMWare and how its a great time saver. So here goes:
Virtualization
Virtualization is, simply put, a form of emulation. Back in the mid 90s emulators became hugely popular because for the first time in history – we had CPU’s powerful enough to emulate other computers at full speed. This was radical because up until that point, you needed special hardware to do that. You had also been limited to emulating legacy systems with no practical business value.
VmWare Workstation is an amazing piece of engineering
Emulation has always been there, even back in the 80s with 16 bit computers. But while it was technically possible, it was more a curiosity than something an office environment would benefit from (unless you used expensive compute boards). We had to wait until the late 90s to see commercial-grade x86 emulation hitting the market, with Virtuozzo releasing Parallels in 1997 and VMWare showing up around 1998. Both of these companies grew out of the data-center culture and academia.
It’s also worth noting that modern CPU’s now support virtualization on hardware level, so when you are “virtualizing” Windows the machine code is not interpreted or JIT compiled – it runs on the same CPU as your real system.
Why does it matter
Virtualization is not just for data-centers and server-farms, it’s also for desktop use. My personal choice was VMWare because I felt their product performed better than the others. But in all fairness it’s been a few years since I compared between systems, so that might be different today.
A screengrab of my desktop, here showing 3 virtual machines running. I have 64 gigabyte memory and these 3 virtual machines consume around 24 gigabytes and uses 17% of the Intel i7 CPU power during compile. It hardly registers on the CPU stats when idle.
VMWare Workstation is a desktop application available for Windows, Linux and OS X. And it allows me to create virtual machines, or “emulations” if you like. The result is that I can run multiple instances of Windows on a single PC. The virtual machines are all sandbox in large hard-disk files, and you have to install Windows or Linux into these virtual systems.
The bonus though is fantastic. Once you have installed an operating-system, you can copy it, move it, do partial cloning (only changes are isolated in new sandboxes) and much, much more. The cloning functionality is incredibly powerful, especially for a developer.
It also gives you something called snap-shot support. A snapshot is, like the word hints to, a copy of whatever state your virtual-machine is in at that point in time. This is a wonderful feature if you remember to use it properly. I try to take snapshots before I install anything, be it larger systems like Delphi, or just utility applications I download. Should something go wrong with the tools your work depends on — you can just roll back to a previous snapshot (!)
A great time saver
Updates to development tools are always awesome, but there are times when things can go wrong. But if you remember to take a snapshot before you install a program, or before you install a component package — should something go wrong, then rolling back to a clean point is reduced to a mouse click.
I mean, imagine you update your development tools right? Suddenly you realize that a component package your software depends on doesn’t work. If you have installed your devtools directly on the metal, you suddenly have a lot of time-consuming work to do:
- Re-install your older devtools
- Re-install your components and fix broken paths
That wont be a problem if you only have 2-3 packages, but I have hundreds of components install on my rig. Just getting my components working can take almost a full work-day, and I’m not exaggerating (!).
With VMWare, I just roll back to when all was fine, and go about my work like nothing happened.
I made a quick, slapdash video to demonstrate how easy VmWare makes my Delphi and JS development. If you are not using virtualization I hope this video at least makes it a bit clearer why so many do.
Click the image to watch the video on YouTube
Five reasons to learn Delphi
A couple of days ago I had a spectacular debate on Facebook. Like most individuals that are active in the IT community, my social media feed is loaded with advertisement for every trending IT concept you can imagine. Lately these adverts have been about machine learning and A.I. Or should I say, companies using those buzzwords to draw unwarranted attention to their products. I haven’t seen A.I used to sell shoes yet, but it’s only a matter of time before it happens.
Like any technology, Cloud is only as powerful as your insight
There is also this thing that: yes, a 14-year-old can put together an A.I chat robot in 15 minutes with product XYZ. But that doesn’t mean he or she understands what is happening beneath the user-interface. Surely the goal must be to teach those kids skills that will benefit them for a lifetime.
Those that know me also know that yes, I have this tendency to say what I mean, even when I really should keep my mouth shut. On the other hand that is also why companies and developers call me, because I will call bullshit and help them avoid it. That’s part of my job, to help individuals and companies that use Delphi to pick the right version for their need, get the components that’s right for their goals – and map out a strategy if they need some input on that. I’ll even dive in and do some code conversion if they need it; goes with the territory.
Normally I just ignore advertizing that put “cloud” or “a.i” in their title, because it’s mostly click-bait designed for non-developers. But for some reason this one particular advert caught my eye. Perhaps it triggered the trauma of being subjected to early Java advertising during the late 90s’s, or maybe it released latent aggression from being psychologically waterboarded by Microsoft Silverlight. Who knows 🙂
The ad was about a Norwegian company that specialize in teaching young students how to become professional developers. You know the “become a guru in 3 weeks” type publisher? What baked my noodle was the fact that they didn’t offer a single course involving archetypical languages, and that they were spinning their material with promises that were simply not true. The only artificial intelligence involved was the advertizing engine at Facebook.
The thing is – the world has more than enough developers on desktop level. The desktop and web market is drowning in developers who has the capacity to download libraries, drop components on a form and hook up to a database. What the world really needs are more developers on archetypical languages. And if you don’t know what that is, then let me just do a quick summary before we carry on.
Archetypal languages
An archetypical programming language is one that is designed around how the computer actually works. As a consequence these languages and toolchains embody several of the following properties:
- Pointers and raw memory access
- Traditional memory management, no garbage collection
- Procedural and object-orientation execution
- Inline assembler
- Little if no external dependencies
- Static linking (embed pre-compiled code)
- Compiled code can operate without an OS infrastructure
- Suitable for kernel, driver, service, desktop, networking and cloud level development
- Compiler that produce machine code for various chipsets
As of writing there are only two archetypical languages (actually 3, but assembly language is chipset specific so we will skip that here), namely C/C++ and Object Pascal. These are the languages you use to write all the other languages with. If you plan on writing your own operating-system from scratch, only C and Pascal is suitable. Which is why these are the only languages that have ever been used for making operating systems.
Delphi is one of the 20 most used programming languages in the world. It ranked as #11 in 2017. Like all rankings it fluctuates depending on season and market changes.
Obviously i’m not suggesting that people learn Delphi or C++ builder to write their own OS – or that you must know assembly to make an invoice system; I’m simply stating that the insight and skill you get from learning Delphi and C/C++, even if all you do is write desktop applications – will make you a better developer on all levels.
Optimistic languages
Optimistic or humanized programming languages, have been around just as long as the archetypical ones. Basic is an optimistic language, C# and Java are optimistic languages, Go and Dart are equally optimistic languages. Script engines like node.js, python and Erlang (if you missed Scott Hanselman’s epic rant on the subject, you are in for a treat) are all optimistic. They are called optimistic because they trade security with functionality; sandboxing the developer from the harsh reality of hardware.
An optimistic language is typically designed to function according to “how human beings would like things to be” (hence the term optimistic). These languages rely heavily on existing infrastructure to even work, and each language tends to focus on specific tasks – only to branch out and become more general purpose over time.
There is nothing wrong with optimistic languages. Except when they are marketed to young students as being somehow superior or en-par with archetypical languages. That is a very dangerous thing to do – because teachers have a responsibility to prepare the students for real life. I can’t even count the number of times I have seen young developers fresh out of college get “that job”, only to realize that the heart of the business, the mission critical stuff, is written in Delphi or C/C++, which they never learned.
People have no idea just how much of the modern world rests on these languages. It is almost alarming how it’s possible to be a developer in 2019 and have a blind spot with regards to these distinctions. Don’t get me wrong, it’s not the student’s fault, quite the opposite. And i’m happy that things are starting to change for the better (more about that further down).
The original full stack
So back to my little encounter; What happened was that I just commented something along the lines of “why not give the kids something that will benefit them for a lifetime”. It was just a drive-by comment on my part, and I should have just ignored it; And no sooner had I pressed enter, when a small army of internet warriors appeared to defend their interpretation of “full stack” in 2019. Oblivious to the fact that the exact same term was used around 1988-ish. I think it was Aztec or SAS-C that coined it. Doesn’t matter.
The original “full stack” holds a very different meaning in traditional development. While I don’t remember if it was Aztec-C or SAS-C, but the full stack was driver to desktop 🙂
Long story short, I ended up having a conversation with these teenagers about how technology has evolved over the past 35 years. Not in theory, but as one that has been a programmer since the C= 64 was released. I also introduced them to archetypal languages and pinpointed the distinction I made above. You cannot compare if you don’t know the difference.
I have no problems with other languages, I use several myself, and my point was simply that: if we are going to teach the next generation of programmers something, then let’s teach them the timeless principles and tools that our eco system rests on. We need to get Delphi and C/C++ back into the curriculum, because that in turn will help the students to become better developers. It doesn’t matter what they end up working with afterwards, because with the fundamental understanding in place they will be better suited. Period.
You will be a better Java developer if you first learn Delphi. You will be a better C# developer if you learn Delphi. Just like nature has layers of complexity, so does computing. And understanding how each layer works and what laws exist there – will have a huge impact on how you write high-level code.
All of this was good and well and the internet warriors seemed a bit confused. They weren’t prepared for an actual conversation. So what started a bit rough ended up as a meaningful, nice dialog.
And speaking of education: I’m happy to say that two universities in Norway now have students using Delphi again. Which is a step in the right direction! People are re-discovering how productive Object-Pascal is, and why the language remains the bread and butter of so many companies around the world.
Piracy, the hydra of problems
What affected me the most during my conversation with these young developers – was that they had almost no relationship to neither Delphi or C/C++. From an educational standpoint that is not just alarming, that is an intellectual emergency. The only knowledge they had of Delphi was hearsay and nonsense.
The source of the misrepresentation is piracy, openly so, of outdated versions that was never designed to run on modern operating systems. With the community edition people can enjoy a modern, high performance Delphi without resorting to illegal activities
But after a while I finally discovered where their information came from! Delphi 7 is being pirated en-mass even to this day. It’s for some strange reason very popular in Asia (most of the torrent IP’s ended up there when I followed up on this). So teenagers download Delphi 7 which is ancient by any standard, and the first thing they experience is incompatibility issues. Which is only to be expected because Delphi 7 was released a long, long time ago. But that’s the impression they are left with after downloading one of these cracked, illegal bundles.
I downloaded one of these “ready to use” bundles to have a closer look, and it contained at least 500 commercial components. You had the full TMS component collection, Developer Express, Remobjects SDK, ImageEN, FastReports, SecureBlackBox, Intraweb — tens of thousands of dollars worth of code. With one very obvious factor: both Delphi 7 and the components involved are severely outdated. Microsoft doesn’t even support Windows XP any more, it was written in the early bronze age.
So the reality of the situation was that these young developers had never seen a modern Delphi in their life. In their minds, Delphi meant Delphi 7 which they could download almost everywhere (which is illegal and riddled with viruses, stay well clear). No wonder there is confusion about the subject (!)
They were very happy to learn about the community edition, so in the end I at least got to wake them up to the awesome features that modern Delphi represents. The community edition has been a fantastic thing; the number of members joining Delphi-Developer on Facebook has nearly doubled since the community edition was released.
A few of the students went over to Embarcadero and downloaded the community edition, and their jaw dropped. They had never seen a development environment like this before!
Give me five good reasons to learn Delphi
In light of this episode, thought I could share five reasons why Delphi and object-pascal remains my primary programming language.
I don’t have any problems dipping into JavaScript, Python or whatever the situation might call for – but when it comes to mission critical data processing and services that needs 24/7 up-time; or embedded solutions where CPU spikes simply cannot be tolerated. It’s Delphi I turn to.
These five reasons are also the same that I gave the teenagers. So here goes.
Great depth and wingspan
Object Pascal, from which Delphi is the trending dialect, is a fantastic language. At heart there is little difference between C/C++ and object pascal in terms of features, but the syntax of object pascal is more productive than C/C++ (IMHO).
Delphi and C++ builder actually share run-time libraries (there are two of them, the VCL which is Windows only, and Firemonkey which is platform independent). Developers often mix and match code between these languages, so components written in Delphi can be used in C++ builder, and libraries written in C can be consumed and linked into your Delphi executable.
One interesting factoid: people imagine Delphi to be old. But the C language is actually 3 years older than pascal. During their time these languages have evolved side by side, and Embarcadero (who makes Delphi and C++ builder) have brought all the interesting features you expect from a modern language into Delphi (things like generics, inline variables, anonymous procedures – it’s all in there). So this myth that Delphi is somehow outdated or unsuitable is just that – a myth.
The eco-system of programming languages
And there is an added bonus! Just like C/C++, Delphi represents a curriculum and lineage that spans decades. Stop and think about that for a second. This is a language that has been evolved to solve technical challenges of every conceivable type for decades. This means that you can put some faith in what the language can deliver.
There are millions of Delphi developers in the world; an estimated 10 millions in fact. The language was ranked #11 on the TIOBI language index; it is under constant development with a clear roadmap and time-line – and is used by large and small companies as the foundation for their business. Even the Norwegian government rely on Delphi. The system that handles healthcare messages for the Norwegian population is pure Delphi. That is data processing for 5.2 million individuals.
Object Pascal has not just stood the test of time, it has innovated it. Just like C/C++ object pascal has a wingspan and depth that reaches from assembler to system services, from database engines to visual desktop application – and from the desktop all the way to Cloud and essential web technology.
So the first good reason to learn Delphi is depth. Delphi covers the native stack, from kernel level drivers to high-speed database engines – to visual desktop applications. It’s also exceptionally well suited for cloud services (both Windows and Linux targets).
Easy to learn
I mention that Delphi is powerful and has the same depth as C/C++, but why then learn Delphi and not C++? Well, the language (object pascal) was especially tailored for readability. It was concluded that the human brain recognized words faster than symbols or glyphs – and thus it’s easier to read complex pascal code rather than complex C code. Individual taste notwithstanding.
Despite its depth, Delphi is easy to learn and fun to master!
Object Pascal is also very declarative, with as little unknown factors as possible. This teaches people to write clean and orderly code.
And perhaps my favorite, a pascal code-file contains both interface and implementation. So you don’t have to write a second .h file which is common under C/C++.
If you already know OOP, be it Java, C#, Rust or whatever – learning Delphi will be a piece of cake. You already know about classes, interfaces, generics, operator overloading – and can pretty much skip forward to memory management, pointers and structures (records in pascal, struct in C).
Swing by Embarcadero Academy and take a course, or head over to Amazon and buy some good books on Delphi. Download the Community Edition of Delphi and you will be up and running in no-time.
Also remember to join Delphi Developer on Facebook, where thousands of active developers talk, help each other and share solutions 24/7.
Target multiple platforms
With Delphi and C++ builder it’s pretty easy to target multiple platforms these days. You can target Android, iOS, OS X, Windows and Linux from a single codebase.
One codebase, multiple targets
I mean, are you going to write one version of your app in Java, a second one in C#, a third one in Objective C and a fourth in Dart? Because that’s the reality you face if plan on using the development tools provided by each operating-system manufacturer. That’s a lot of time, money and effort just to push your product out the door.
With Delphi you can hit all platforms at once, native code, reducing your time to market and ROI. People use Delphi for a reason.
You will also enjoy great performance from the LLVM optimized code Delphi emits on mobile platforms.
Rich codebase
The benefit of age is often said to be wisdom; I guess the computing equivalent is a large and rich collection of components, libraries and ad-hoc code that you can drop into your own projects or just study.
You can google just about any subject, and there will be code for Delphi. Github, BitBucket and Torry’s Delphi pages are packed with open-source frameworks covering everything from compiler cores, midi interfaces, game development to multi-threaded, machine clustered server solutions. Once you start looking, you will find it.
There is a rich constellation of code, components and libraries for Delphi and C++ builder around the internet. Also remember dedicated sites like Torry’s
There is also a long list of technology partners that produce components and libraries for Delphi – and like mentioned earlier, you can link in C compiled code once you learn the ropes.
Oh, and when I mentioned databases earlier I wasnt just talking about the traditional databases. Delphi got you covered with those, no worries — im also talking about writing a database engine from scratch. There are several database engines that are implemented purely in Delphi. ElevateDB is one example.
Delphi also ships with Interbase and Interbase-light (embedded and mobile) so you have easy access to data storage solutions. There is also FireFAC that allows you to connect directly with established databases — and again, a wealth of free and commercial solutions.
Speed and technique
What I love about Delphi and C++ is that your code, or the way you write code, directly impacts your results. The art of optimization is rarely a factor in some of the new, optimistic languages. But in a native language you get to use traditional techniques that are time-less, or perhaps more interesting: explore ways of achieving the same with less.
As a native language Delphi and C/C++ produce fast executables. But I love how there is always room for your own techniques, your own components and your own libraries.
Techniques, like math, is timeless
Need to write a system driver? Well, suddenly speed becomes a very important factor. A garbage collector can be a disaster on that level, because it will kick-in on interval and cause CPU spikes. Perhaps you want to write a compiler, or need a solid scripting engine? How about linking the V8 JavaScript engine directly into your programs? All of this is quite simple with Delphi.
So with Delphi I get the best of both worlds, I get to use the scalpel when the needs are delicate, and I get the chain-saw to cut through tedious work. Things like property bindings are a god sent. This is a techniques where you can visually bind properties of any component together, almost like events, and create cause and effect chains. So if a value changes on a bound property, that triggers whatever is bound, and so on and so on — pretty awesome!
So you can create a complete database application, with grid and navigation, without writing a single line of code. That was just one simple example, you can do so much more out of the box – and it saves you so much time.
Yet when you really need to write high performance code, or build that killer framework that will set your company apart from the rest — you have that freedom!
So if you havent checked out RAD Studio, head over to Embarcadero and download a free trial. You will be amazed and realize just why Delphi and C++ builder are loved by so many.
Delphi “real life” webinars
I got some great news for everyone!
For a while now we have been planning some Delphi community webinars. This will be a monthly webinar that has a slightly different format than what people are used to. The style of webinar will be live, laid back and with focus on real-life solutions that already exists, or that is being developed – talking directly to the developers and MVP’s involved.
There is so much cool happening in the Delphi, C++ builder and Sencha scene that I hardly know where to begin. But what better way to spread the good news than to talk directly with the people building the components, publishing the software, doing that book or rolling the frameworks?
In the group Delphi Developer on Facebook we have a very laid back style, one I hope to transpose onto the webinars. We keep things clean, have clear rules and the atmosphere is friendly and easy-going. There is room for jokes and off topic posts on the weekends, but above all: we are active, solution oriented developers.
Delphi Developer, although being small compared to the 6.5 million registered Delphi developers in the world (estimated object pascal use is closer to 10 million when factoring in alternative compilers), just reached 8000 active members. The growth rate for membership into our little corner of the world has really picked-up speed after the community edition. Seriously, it’s phenomenal to be a part of this. It’s more than doubled since 2017.
So there has never been a better time to do webinars on Delphi than right now 🙂
Making waves
Delphi has so much to offer
Two weeks ago I was informed that Delphi is once again being used by one of the largest Norwegian universities (!). That was an epic moment, because that is something we have worked hard to realize. I have been blogging, teaching and doing pro-bono work for a decade now to get the ball rolling – and seeing the community revitalize itself is spectacular!
I work like mad every day to help companies with strategies involving Delphi, showing them how they can use Delphi to strengthen their existing infrastructure; I connect developers to employers, do casual, drive-by headhunting, talk to component vendors — but education and awareness is what it’s all about. Your toolbox is only as useful as your knowledge of the tools. If you don’t know how or what a tool is, well then you probably wont use it much.
Making new developers aware of what Delphi is and what it can do is at the heart of this. Especially developers that work with other languages. The reality of 2019 is that companies use several languages to build their infrastructure, and it’s important that they understand how Delphi can co-exist and benefit their existing investment. So a fair share of my time is about educating developers from other eco-systems. Most of them are not prepared for the great depth and wingspan object pascal has, and are flabbergasted when the full scope of the product hits them. Only C++ and object pascal scales from kernel to cloud. That’s the real full stack right there.
Delphi: The secret in the sause
I keep up with whats happening in many different parts of development, and one of those is node.js and webassembly. Since everyone was strutting their stuff I figured I could as well, so I posted some videos about the Quartex Web Desktop I have been working on in my spare time (a personal project done in Object Pascal and compiled to JavaScript).
The result? The node.js groups on Facebook went nuts! Within minutes of posting I was bombarded by personal message requests, friend requests and even a marriage proposal. All of it from young web developers wanting to know “my secrets”.
Well, the secret is Delphi. I mean, I can sugarcoat it as much as I want, but without Delphi none of the auxiliary tools I use or made would exist. They are all made with Delphi – for Delphi developers. Smart Mobile Studio, the QTX framework, my libraries and tools – none of them would have seen the light of day if I never learned Delphi.
Node developers could not believe their eyes nor ears when they learned that this system was coded in Object Pascal, using a “off the shelf” compiler that is 100% Delphi; DWScript and Smart Mobile Studio is a pretty common addition to Delphi developers toolbox in 2019
What I’m trying to convey to young developers especially, is that if you take the time to learn Delphi, you can pick from so many third-party associated technologies that will help you create incredible software. ImageEN, AToZed, DevEx, TMS Component Suite, Greatis Software, FastReports, DWScript, Smart Mobile Studio; and that is just the tip of the iceberg (not to mention the amazing products by Boian Mitov, talk about powerful solutions!). As a bonus you have thousands of free components and units on Github, Torry’s and other websites.
That’s a pretty strong case. We are talking real-life business here, not dorm-room philosophical idealism. You have 800.000 receipts on average hitting your servers on a daily basis — and you have 20.000 cash machines in Norway alone that must function 24/7; you have no room for cpu spikes on the embedded board, nor can you tolerate latency problems or customers start walking. And you need it up and running yesterday. I can tell you right now having experienced that exact scenario, that had we used any other tool than Delphi – it would have sunk the company.
The point? After posting some videos and chatting a bit with the node.js devs, Delphi Developer got infused with a sizable chunk of young node.js developers eager to learn more about this “Delphi thing”. And they will become better node developers for it.
EDIT: I started this day (01.02.19) with a call from a university student. He was fed up with Java and C# because he wanted to learn native programming. He had noticed the node.js post and became curious. So I set him up with the community edition of both Delphi and C++ builder. When he masters the basics I will introduce him to inline assembler. There is a gap in modern education where Delphi used to sit, and no matter how much they try to fill it, bytecodes can’t replace solid knowledge of how a computer actually works.
So indeed! These webinars will be very fun to make. We got so many fantastic developers to invite, techniques to explore, components to demo and room for questions! The hard part right now is actually picking topics, because we have so much to choose from!
For example, did you know there is TV channel that is operated using Delphi software? It’s been running without a glitch for decades. Rock solid and high performance. How cool is that! Talk about real-life solution. Delphi is everywhere.
I’ll get back to you with more information in due time ~ Cheers!
Amibian.js under the hood
Amibian.js is gaining momentum as more and more developers, embedded systems architects, gamers and retro computer enthusiasts discover the project. And I have to admit I’m pretty stoked about what we are building here myself!
In a life-preserver no less 😀
But, with any new technology or invention there are two common traps that people can fall into: The first trap is to gravely underestimate a technology. JavaScript certainly invites this, because only a decade ago the language was little more than a toy. Since then JavaScript have evolved to become the most widely adopted programming language in the world, and runtime engines like Google’s V8 runs JavaScript almost as fast as compiled binary code (“native” means machine code, like that produced by a C/C++ compiler, Pascal compiler or anything else that produces programs that run under Linux or Windows).
It takes some adjustments, especially for traditional programmers that havent paid attention to where browsers have gone – but long gone are the days of interpreted JavaScript. Modern JavaScript is first parsed, tokenized and compiled to bytecodes. These bytecodes are then JIT compiled (“just in time”, which means the compilation takes place inside the browser) to real machine-code using state of the art techniques (LLVM). So the JavaScript of 2018 is by no means the JavaScript of 2008.
The second trap you can fall into – is to exaggerate what a new technology can do, and attach abilities and expectations to a product that simply cannot be delivered. It is very important to me that people don’t fall into either trap, and that everyone is informed about what Amibian.js actually is and can deliver – but also what it wont deliver. Rome was not built-in a day, and it’s wise to study all the factors before passing judgement.
I have been truly fortunate that people support the project financially via Patreon, and as such I feel it’s my duty to document and explain as much as possible. I am a programmer and I often forget that not everyone understands what I’m talking about. We are all human and make mistakes.
Hopefully this post will paint a clearer picture of Amibian.js and what we are building here. The project is divided into two phases: first to finish Amibian.js itself, and secondly to write a Visual Studio clone that runs purely in the browser. Since it’s easy to mix these things up, I’m underlining this easy – just in case.
What the heck is Amibian.js?
Amibian.js is a group of services and libraries that combined creates a portable operating-system that renders to HTML5. A system that was written using readily available web technology, and designed to deliver advanced desktop functionality to web applications.
The services that make up Amibian.js was designed to piggyback on a thin Linux crust, where Linux deals with the hardware, drivers and the nitty-gritty we take for granted. There is no point trying to write a better kernel in 2018, because you are never going to catch up with Linus Torvalds. It’s must more interesting to push modern web technology to the absolute limits, and build a system that is truly portable and distributed.
Above: Amibian.js is created in Smart Pascal and compiled to JavaScript
The service layer is written purely in node.js (JavaScript) which guarantees the same behavior regardless of host platform. One of the benefits of using off-the-shelves web technology is that you can physically copy the whole system from one machine to the other without any changes. So if you have a running Amibian.js system on your x86 PC, and copy all the files to an ARM computer – you dont even have to recompile the system. Just fire up the services and you are back in the game.
Now before you dismiss this as “yet another web mockup” please remember what I said about JavaScript: the JavaScript in 2018 is not the JavaScript of 2008. No other language on the planet has seen as much development as JavaScript, and it has evolved from a “browser toy” – into the most important programming language of our time.
So Amibian.js is not some skin-deep mockup of a desktop (lord knows there are plenty of those online). It implements advanced technologies such as remote filesystem mapping, an object-oriented message protocol (Ragnarok), RPCS (remote procedure call invocation stack), video codec capabilities and much more — all of it done with JavaScript.
In fact, one of the demos that Amibian.js ships with is Quake III recompiled to JavaScript. It delivers 120 fps flawlessly (browser is limited to 60 fps) and makes full use of standard browser technologies (WebGL).
Click on picture above to watch Amibian.js in action on YouTube
So indeed, the JavaScript we are talking about here is cutting edge. Most of Amibian.js is compiled as “Asm.js” which means that the V8 runtime (the code that runs JavaScript inside the browser, or as a program under node.js) will JIT compile it to highly efficient machine-code.
Which is why Amibian.js is able to do things that people imagine impossible!
Ok, but what does Amibian.js consist of?
Amibian.js consists of many parts, but we can divide it into two categories:
- A HTML5 desktop client
- A system server and various child processes
These two categories have the exact same relationship as the X desktop and the Linux kernel. The client connects to the server, invokes procedures to do some work, and then visually represent the response This is identical to how the X desktop calls functions in the kernel or one of the Linux libraries. The difference between the traditional, machine code based OS and our web variation, is that our version doesn’t have to care about the hardware. We can also assign many different roles to Ambian.js (more about that later).
Enjoying other cloud applications is easy with Amibian.js, here is Plex, a system very much based on the same ideas as Amibian.js
And for the record: I’m trying to avoid a bare-metal OS, otherwise I would have written the system using a native programming language like C or Object-Pascal. So I am not using JavaScript because I lack skill in native languages, I am using JavaScript because native code is not relevant for the tasks Amibian.js solves. If I used a native back-end I could have finished this in a couple of months, but a native server would be unable to replicate itself between cloud instances because chipset and CPU would be determining factors.
The Amibian.js server is not a single program. The back-end for Amibian.js consists of several service applications (daemons on Linux) that each deliver specific features. The combined functionality of these services make up “the amibian kernel” in our analogy with Linux. You can think of these services as the library files in a traditional system, and programs that are written for Amibian.js can call on these to a wide range of tasks. It can be as simple as reading a file, or as complex as registering a new user or requesting admin rights.
The greatest strength of Amibian.js is that it’s designed to run clustered, using as many CPU cores as possible. It’s also designed to scale, meaning that it will replicate itself and divide the work between different instances. This is where things get’s interesting, because an Amibian.js cluster doesn’t need the latest and coolest hardware to deliver good performance. You can build a cluster of old PC’s in your office, or a handful of embedded boards (ODroid XU4, Raspberry PI’s and Tinkerboard are brilliant candidates).
But why Amibian.js? Why not just stick with Linux?
That is a fair question, and this is where the roles I mentioned above comes in.
As a software developer many of my customers work with embedded devices and kiosk systems. You have companies that produce routers and set-top boxes, NAS boxes of various complexity, ticket systems for trains and busses; and all of them end up having to solve the same needs.
What each of these manufacturers have in common, is the need for a web desktop system that can be adapted for a specific program. Any idiot can write a web application, but when you need safe access to the filesystem, unified API’s that can delegate signals to Amazon, Azure or your company server, things suddenly get’s more complicated. And even when you have all of that, you still need a rock solid application model suitable for distributed computing. You might have 1 ticket booth, or 10.000 nation wide. There are no systems available that is designed to deal with web-technology on that scale. Yet 😉
Let’s look at a couple of real-life scenarios that I have encountered, I’m confident you will recognize a common need. So here are some roles that Amibian.js can assume and help deliver a solution rapidly. It also gives you some ideas of the economic possibilities.
Updated: Please note that we are talking javascript here, not native code. There are a lot of native solutions out there, but the whole point here is to forget about CPU, chipset and target and have a system floating on top of whatever is beneath.
- When you want to change some settings on your router – you login to your router. It contains a small apache server (or something similar) and you do all your maintenance via that web interface. This web interface is typically skin-deep, annoying to work with and a pain for developers to update since it’s connected to a native apache module which is 100% dependent on the firmware. Each vendor end up re-inventing the wheel over and over again.
- When you visit a large museum notice the displays. A museum needs to display multimedia, preferably on touch capable devices, throughout the different exhibits. The cost of having a developer create native applications that displays the media, plays the movies and gives visual feedback is astronomical. Which is why most museums adopt web technology to handle media presentation and interaction. Again they re-invent the wheel with varying degree of success.
- Hotels have more or less the exact same need but on a smaller scale, especially the larger hotels where the lobby have information booths, and each room displays a web interface via the TV.
- Shopping malls face the same challenge, and depending on the size they can need anything from a single to a hundred nodes.
- Schools and education spend millions on training software and programming languages every year. Amibian.js can deliver both and the schools would only pay for maintenance and adaptation – the product itself is free. Kids get the benefit of learning traditional languages and enjoying instant visual feedback! They can learn Basic, Pascal, JavaScript and C. I firmly believe that the classical languages will help make them better programmers as they evolve.
You are probably starting to see the common denominator here?
They all need a web-based desktop system, one that can run complex HTML5 based media applications and give them the same depth as a native operating-system; Which is pretty hard to achieve with JavaScript alone.
Amibian.js provides a rich foundation of more than 4000 classes that developers can use to write large, complex and media rich applications (see Smart Mobile Studio below). Just like Linux and Windows provides a wealth of libraries and features for native application development – Amibian.js aims to provide the same for cloud and embedded systems.
And as the name implies, it has roots in the past with the machine that defined multimedia, namely the Commodore Amiga. So the relation is more than just visually, Amibian.js uses the same system architecture – because we believe it’s one of the best systems ever designed.
If JavaScript is so poor, why should we trust you to deliver so much?
First of all I’m not selling anything. It’s not like this project is something that is going to make me a ton of cash. I ask for support during the development period because I want to allocate proper time for it, but when done Amibian.js will be free for everyone (LGPL). And I’m also writing it because it’s something that I need and that I havent seen anywhere else. I think you have to write software for yourself, otherwise the quality wont be there.
Secondly, writing Amibian.js in raw JavaScript with the same amount of functions and depth would take years. The reason I am able to deliver so much functionality quickly, is because I use a compiler system called Smart Mobile Studio. This saves months and years of development time, and I can use all the benefits of OOP.
Prior to starting the Amibian.js project, I spent roughly 9 years creating Smart Mobile Studio. Smart is not a solo project, many individuals have been involved – and the product provides a compiler, IDE (editor and tools), and a vast run-time library of pre-made classes (roughly 4000 ready to use classes, or building-blocks).
Writing large-scale node.js services in Smart is easy, fun and powerful!
Unlike other development systems, Smart Mobile Studio compiles to JavaScript rather than machine-code. We have spent a great deal of time making sure we could use proper OOP (object-oriented programming), and we have spent more than 3 years perfecting a visual application framework with the same depth as the VCL or FMX (the core visual frameworks for C++ builder and Delphi).
The result is that I can knock out a large application that a normal JavaScript coder would spend weeks on – in a single day.
Smart Mobile Studio uses the object-pascal language, a dialect which is roughly 70% compatible with Delphi. Delphi is exceptionally well suited for writing large, data driven applications. It also thrives for embedded systems and low-level system services. In short: it’s a lot easier to maintain 50.000 lines of object pascal code, than 500.000 lines of JavaScript code.
Amibian.js, both the service layer and the visual HTML5 client application, is written completely using Smart Mobile Studio. This gives me as the core developer of both systems a huge advantage (who knows it better than the designer right?). I also get to write code that is truly OOP (classes, inheritance, interfaces, virtual and abstract methods, partial classes etc), because our compiler crafts something called a VMT (virtual method table) in JavaScript.
Traditional JavaScript doesn’t have OOP, it has something called prototypes. With Smart Pascal I get to bring in code from the object-pascal community, components and libraries written in Delphi or Freepascal – which range in the hundreds of thousands. Delphi alone has a massive library of code to pick from, it’s been a popular toolkit for ages (C is 3 years older than pascal).
But how would I use Amibian.js? Do I install it or what?
Amibian.js can be setup and used in 4 different ways:
- As a true desktop, booting straight into Amibian.js in full-screen
- As a cloud service, accessing it through any modern browser
- As a NAS or Kiosk front-end
- As a local system on your existing OS, a batch script will fire it up and you can use your browser to access it on https://127.0.0.1:8090
So the short answer is yes, you install it. But it’s the same as installing Chrome OS. It’s not like an application you just install on your Linux, Windows or OSX box. The whole point of Amibian.js is to have a platform independent, chipset agnostic system. Something that doesn’t care if you using ARM, x86, PPC or Mips as your CPU of preference. Developers will no doubt install it on their existing machines, Amibian.js is non-intrusive and does not affect or touch files outside its own eco-system.
But the average non-programmer will most likely setup a dedicated machine (or several) or just deploy it on their home NAS.
The first way of enjoying Amibian.js is to install it on a PC or ARM device. A disk image will be provided for supporters so they can get up and running ASAP. This disk image will be based on a thin Linux setup, just enough to get all the drivers going (but no X desktop!). It will start all the node.js services and finally enter a full-screen web display (based on Chromium Embedded) that renders the desktop. This is the method most users will prefer to work with Amibian.js.
The second way is to use it as a cloud service. You install Amibian.js like mentioned above, but you do so on Amazon or Azure. That way you can login to your desktop using nothing but a web browser. This is a very cost-effective way of enjoying Amibian.js since renting a virtual instance is affordable and storage is abundant.
The third option is for developers. Amibian.js is a desktop system, which means it’s designed to host more elaborate applications. Where you would normally just embed an external website into an IFrame, but Amibian.js is not that primitive. Hosting external applications requires you to write a security manifest file, but more importantly: the application must interface with the desktop through the window’s message-port. This is a special object that is sent to the application as a hand-shake, and the only way for the application to access things like the file-system and server-side functionality, is via this message-port.
Calling “kernel” level functions from a hosted application is done purely via the message-port mentioned above. The actual message data is JSON and must conform to the Ragnarok client protocol specification. This is not as difficult as it might sound, but Amibian.js takes security very seriously – so applications trying to cause damage will be promptly shut down.
You mention hosted applications, do you mean websites?
Both yes and no: Amibian.js supports 3 types of applications:
- Ordinary HTML5/JS based applications, or “websites” as many would call them. But like I talked about above they have to establish a dialog with the desktop before they can do anything useful.
- Hybrid applications where half is installed as a node.js service, and the other half is served as a normal HTML5 app. This is the coolest program model, and developers essentially write both a server and a client – and then deploy it as a single package.
- LDEF compiled bytecode applications, a 68k inspired assembly language that is JIT compiled by the browser (commonly called “asm.js”) and runs extremely fast. The LDEF virtual machine is a sub-project in Amibian.js
The latter option, bytecodes, is a bit like Java. A part of the Amibian.js project is a compiler and runtime system called LDEF.
Above: The Amibian.js LDEF assembler, here listing opcodes + disassembling a method
The first part of the Amibian.js project is to establish the desktop and back-end services. The second part of the project is to create the worlds first cloud based development platform. A full Visual Studio clone if you like, that allows anyone to write cloud, mobile and native applications directly via the browser (!)
Several languages are supported by LDEF, and you can write programs in Object Pascal, Basic and C. The Basic dialect is especially fun to work with, since it’s a re-implementation of BlitzBasic (with a lot of added extras). Amiga developers will no doubt remember BlitzBasic, it was used to create some great games back in the 80s and 90s. It’s well suited for games and multimedia programming and above all – very easy to learn.
More advanced developers can enjoy Object Pascal (read: Delphi) or a sub-set of C/C++.
And please note: This IDE is designed for large-scale applications, not simple snippets. The ultimate goal of Amibian.js is to move the entire development cycle to the cloud and away from the desktop. With Amibian.js you can write a cool “app” in BlitzBasic, run it right in the browser — or compile it server-side and deploy it to your Android Phone as a real, natively compiled application.
So any notion of a “mock desktop for HTML” should be firmly put to the side. I am not playing around with this product and the stakes are very real.
But why don’t you just use ChromeOS?
There are many reasons, but the most important one is chipset independence. Chrome OS is a native system, meaning that it’s core services are written in C/C++ and compiled to machine code. The fundamental principle of Amibian.js is to be 100% platform agnostic, and “no native code allowed”. This is why the entire back-end and service layer is targeting node.js. This ensures the same behavior regardless of processor or host system (Linux being the default host).
Node.js has the benefit of being 100% platform independent. You will find node.js for ARM, x86, Mips and PPC. This means you can take advantage of whatever hardware is available. You can even recycle older computers that have lost mainstream support, and use them to run Amibian.js.
A second reason is: Chrome OS might be free, but it’s only as open as Google want it to be. ChromeOS is not just something you pick up and start altering. It’s dependence on native programming languages, compiler toolchains and a huge set of libraries makes it extremely niche. It also shields you utterly from the interesting parts, namely the back-end services. It’s quite frankly boring and too boxed in for any practical use; except for Google and it’s technology partners that is.
I wanted a system that I could move around, that could run in the cloud, on cheap SBC’s. A system that could scale from handling 10 users to 1000 users – a system that supports clustering and can be installed on multiple machines in a swarm.
A system that anyone with JavaScript knowledge can use to create new and exciting systems, that can be easily expanded and serve as a foundation for rich media applications.
What is this Amiga stuff, isn’t that an ancient machine?
In computing terms yes, but so is Unix. Old doesn’t automatically mean bad, it actually means that it’s adapted and survived challenges beyond its initial design. While most of us remember the Amiga for its games, I remember it mainly for its elegant and powerful operating-system. A system so flexible that it’s still in use around the world – 33 years after the machine hit the market. That is quite an achievement.
The original Amiga OS, not bad for a 33-year-old OS! It was and continues to be way ahead of everyone else. A testament to the creativity of its authors
Amibian.js as the name implies, borrows architectural elements en-mass from Amiga OS. Quite simply because the way Amiga OS is organized and the way you approach computing on the Amiga is brilliant. Amiga OS is much more intuitive and easier to understand than Linux and Windows. It’s a system that you could learn how to use fully with just a couple of days exploring; and no manuals.
But the similarities are not just visual or architectural. Remember I wrote that hosted applications can access and use the Amibian.js services? These services implement as much of the original ROM Kernel functions as possible. Naturally I can’t port all of it, because it’s not really relevant for Amibian.js. Things like device-drivers serve little purpose for Amibian.js, because Amibian.js talks to node.js, and node talks to the actual system, which in turn handles hardware devices. But the way you would create windows, visual controls, bind events and create a modern, event-driven application has been preserved to the best of my ability.
But how does this thing boot? I thought you said server?
If you have setup a dedicated machine with Amibian.js then the boot sequence is the same as Linux, except that the node.js services are executed as background processes (daemons or services as they are called), the core server is initialized, and then a full-screen HTML5 view is set up that shows the desktop.
But that is just for starting the system. Your personal boot sequence which deals with your account, your preferences and adaptations – that boots when you login to the system.
When you login to your Amibian.js account, no matter if it’s just locally on a single PC, a distributed cluster, or via the browser into your cloud account — several things happen:
- The client (web-page if you like) connects to the server using WebSocket
- Login is validated by the server
- The client starts loading preferences files via the mapped filesystem, and then applies these to the desktop.
- A startup-sequence script file is loaded from your account, and then executed. The shell-script runtime engine is built into the client, as is REXX execution.
- The startup-script will setup configurations, create symbolic links (assigns), mount external devices (dropbox, google drive, ftp locations and so on)
- When finished the programs in the ~/WbStartup folder are started. These can be both visual and non-visual.
As you can see Amibian.js is not a mockup or “fake” desktop. It implements all the advanced features you expect from a “real” desktop. The filesystem mapping is especially advanced, where file-data is loaded via special drivers; drivers that act as a bridge between a storage service (a harddisk, a network share, a FTP host, Dropbox or whatever) and the desktop. Developers can add as many of these drivers as they want. If they have their own homebrew storage system on their existing servers, they can implement a driver for it. This ensures that Amibian.js can access any storage device, as long as the driver conforms to the driver standard.
In short, you can create, delete, move and copy files between these devices just like you do on Windows, OSX or the Linux desktop. And hosted applications that run inside their own window can likewise request access to these drivers and work with the filesystem (and much more!).
Wow this is bigger than I thought, but what is this emulation I hear about? Can Amibian.js really run actual programs?
Amibian.js has a JavaScript port of UAE (Unix Amiga Emulator). This is a fork of SAE (scripted Amiga Emulator) that has been heavily optimized for web. Not only is it written in JavaScript, it performs brilliantly and thus allows us to boot into a real Amiga system. So if you have some floppy-images with a game you love, that will run just fine in the browser. I even booted a 2 gigabyte harddisk image 🙂
But Amiga emulation is just the beginning. More and more emulators are ported to JavaScript; you have Nes, SNes, N64, PSX I & II, Sega Megadrive and even a NEO GEO port. So playing your favorite console games right in the browser is pretty straight forward!
But the really interesting part is probably QEmu. This allows you to run x86 instances directly in the browser too. You can boot up in Windows 7 or Ubuntu inside an Amibian.js window if you like. Perhaps not practical (at this point) but it shows some of the potential of the system.
I have been experimenting with a distributed emulation system, where the emulation is executed server-side, and only the graphics and sound is streamed back to the Amibian.js client in real-time. This has been possible for years via Apache Guacamole, but doing it in raw JS is more fitting with our philosophy: no native code!
I heard something about clustering, what the heck is that?
Remember I wrote about the services that Amibian.js has? Those that act almost like libraries on a physical computer? Well, these services don’t have to be on the same machine — you can place them on separate machines and thus its able to work faster.
Above: The official Amibian.js cluster, 4 x ODroid XU4s SBC’s in a micro-rack
A cluster is typically several computers connected together, with the sole purpose of having more CPU cores to divide the work on. The cool thing about Amibian.js is that it doesn’t care about the underlying CPU. As long as node.js is available it will happily run whatever service you like – with the same behavior and result.
The official Amibian.js cluster consists of 5 ODroid XU4/S SBC (single board computers). Four of these are so-called “headless” computers, meaning that they don’t have a HDMI port – and they are designed to be logged into and software setup via SSH or similar tools. The last machine is a ODroid XU4 with a HDMI out port, which serves as “the master”.
The architecture is quite simple: We allocate one whole SBC for a single service, and allow the service to copy itself to use all the CPU cores available (each SBC has 8 CPU cores). With this architecture the machine that deals with the desktop clients don’t have to do all the grunt work. It will accept tasks from the user and hosted applications, and then delegate the tasks between the 4 other machines.
Note: The number of SBC’s is not fixed. Depending on your use you might not need more than a single SBC in your home setup, or perhaps two. I have started with 5 because I want each part of the architecture to have as much CPU power as possible. So the first “official” Amibian.js setup is a 40 core monster shipping at around $250.
But like mentioned, you don’t have to buy this to use Amibian.js. You can install it on a single spare X86 PC you have, or daisy chain a couple of older PC’s on a switch for the same result.
Why Headless? Don’t you need a GPU?
The headless SBC’s in the initial design all have GPU (graphical processing unit) as well as audio capabilities. What they lack is GPIO pins and 3 additional USB ports. So each of the nodes on our cluster can handle graphics at blistering speed — but that is ultimately not their task. They serve more as compute modules that will be given tasks to finish quickly, while the main machine deals with users, sessions, traffic and security.
The 40 core cluster I use has more computing power than northern europe had in the early 80s, that’s something to think about. And the pricetag is under $300 (!). I dont know about you but I always wanted a proper mainframe, a distributed computing platform that you can login to and that can perform large tasks while I do something else. This is as close as I can get on a limited budget, yet I find the limitations thrilling and fun!
Part of the reason I have opted for a clustered design has to do with future development. While UAE.js is brilliant to emulate an Amiga directly in the browser – a more interesting design is to decouple the emulation from the output. In other words, run the emulation at full speed server-side, and just stream the display and sounds back to the Amibian.js display. This would ensure that emulation, of any platform, runs as fast as possible, makes use of multi-processing (read: multi threading) and fully utilize the network bandwidth within the design (the cluster runs on its own switch, separate from the outside world-wide-web).
I am also very interested in distributed computing, where we split up a program and run each part on different cores. This is a topic I want to investigate further when Amibian.js is completed. It would no doubt require a re-design of the LDEF bytecode system, but this something to research later.
Will Amibian.js replace my Windows box?
That depends completely on what you use Windows for. The goal is to create a self-sustaining system. For retro computing, emulation and writing cool applications Amibian.js will be awesome. But Rome was not built-in a day, so it’s wise to be patient and approach Amibian.js like you would Chrome OS. Some tasks are better suited for native systems like Linux, but more and more tasks will run just fine on a cloud desktop like Amibian.js.
Until the IDE and compilers are in place after phase two, the system will be more like an embedded OS. But when the LDEF compiler and IDE is in place, then people will start using it en-mass and produce applications for it. It’s always a bit of work to reach that point and create critical mass.
Object Pascal is awesome, but modern, native development systems are quite demanding
My personal need has to do with development. Some of the languages I use installs gigabytes onto my PC and you need a full laptop to access them. I love Amibian.js because I will be able to work anywhere in the world, as long as a browser and normal internet line is available. In my case I can install a native compiler on one of the nodes in the cluster, and have LDEF emit compatible code; voila, you can build app-store ready applications from within a browser environment.
I also love that I can set-up a dedicated platform that runs legacy applications, games – and that I can write new applications and services using modern, off the shelve languages. And should a node in the cluster break down, I can just copy the whole system over to a new, affordable SBC and keep going. No super expensive hardware to order, no absurd hosting fees, and finally a system that we all can shape and use in a plethora of systems. From a fully fledged desktop to a super advanced NAS or Router that use Amibian.js to give it’s customers a fantastic experience.
And yes, I get to re-create the wonderful reality of Amiga OS without the absurd egoism that dominates the Amiga owners to this day. I don’t even know where to begin with the present license holders – and I am so sick of the drama that rolling my own seemed the only reasonable path forward.
Well — I hope this helps clear up any misconceptions about Amibian.js, and that you find this as interesting as I do. As more and more services are pushed cloud-side, the more relevant Amibian.js will become. It is perfect as a foundation for large-scale applications, embedded systems — and indeed, as a solo platform running on embedded devices!
I cant wait to finish the services and cluster this sucker on the ODroid rack!
If you find this project interesting, head over to my Patreon website and get involved! I could really use your support, even if it’s just a $5 “high five”. Visit the project at: http://www.patreon.com/quartexNow
Admin woes on Delphi Developer
For well over 10 years I have been running different interest groups on Facebook. While Delphi Developer is without a doubt the one that receives most attention from myself and my fellow moderators, I also run the Quartex Components group and lately, Amiga Disrupt. The latter dedicated to my favorite hobby, namely retro computing.
I have to say, it’s getting harder to operate these groups under the current Facebook regime. I applaud them for implementing a moral codex, that is both fair and good, but that also means that their code must be able to distinguish between random acts of hate and bullying, and moderator operations.
A couple of days ago I posted an update picture from Amibian.js. This is a picture of my vmware development platform, with pascal code, node.js and the HTML5 desktop running. You would have be completely ignorant of technology to not recognize the picture as having to do with software development.
This picture was flagged as hateful, and was enough to get an admin’s account frozen for 30 days
Sadly facebook contains all sorts of people, and for some reason even grown men will get into strange, ideological debates about what constitutes retro-computing. In this case the user was a die-hard original-amiga fan, who on seeing my post about amibian.js went on a spectacular rant. Listing in alphabetical and chronological order, the depths of depravity that people have stooped to in implementing 68k as Javascript.
Well, I get 2-3 of these comments a week and the rules for the group is crystal clear: if you post comments like that, or comments that are racist, hateful or otherwise regarded as a provocative to the general group standard — you are given a single warning and then you are out.
So I gave him a warning that such comments are not welcome; He immediately came back with a even worse response – and that was the end of that.
But before I managed to kick the user, he reported a picture of Amibian as hateful. Again, we are talking about a screen-dump from VMWare with pascal code. No hate, no poor choice of images – nothing that would violate ordinary Facebook standards.
The result? Facebook has now frozen my account for 30 days (!)
Well I’m not even going to bother being upset, because this is not the first time. When people seem to willfully seek out conflict, only to use the FB’s reporting tools as weapons of revenge — well, there is not much I can do.
Anyways, Gunnar, Glenn, Peter and Dennis got you covered – and I’ll see you in a month. I think it’s time i contact FB in Oslo and establish separate management profiles.
