Archive
Vector Containers For Delphi and FPC
Edit: Version 1.0.1 has been released, with a ton of powerful features. Read about it here and grab your fork: https://jonlennartaasenden.wordpress.com/2020/04/13/qtx-framework-for-delphi-and-fpc-is-available-on-bitbucket/
If you have been looking at C++ and envied them their std::vector classes, wanting the same for Delphi or being able to access untyped memory using a typed-view (basically turning a buffer into an array of <T>) then I have some good news for you!
Vector containers, unified storage model and typed views are just some of the highlights of my vector-library. I did an article on the subject at the Embarcadero community website, so head over and read up on how you can enjoy these features in your Delphi application!
I also added FreePascal support, so that the library can be used with TMS Web Framework.

Head over to the Embarcadero Community website to read the full article
C/C++ porting, QTX and general status
C is a language that I used to play around with a lot back in the Amiga days. I think the last time I used a C compiler to write a library must have been in 1992 or something like that? I held on to my Amiga 1200 for as long as i could – but having fallen completely in love with Pascal, I eventually switched to x86 and went down the Turbo Pascal road.
Lately however, C++ developers have been asking for their own Developer group on Facebook. I run several groups on Facebook in the so-called “developer” family. So you have Delphi Developer, FPC Developer, Node.JS Developer and now – C++Builder developer. The groups more or less tend to themselves, and the node.js and FPC groups are presently being seeded (meaning, that the member count is being grown for a period).
The C++Builder group however, is having the same activity level as the Delphi group almost, thanks to some really good developers that post links, tips and help solve questions. I was also fortunate enough to have David Millington come on the Admin team. David is leading the C++Builder project, so his insight and knowledge of both language and product is exemplary. Just like Jim McKeeth, he is a wonderful resource for the community and chime in with answers to tricky questions whenever he has time to spare.
Getting back in the saddle
Having working some 30 years with Pascal and Object Pascal, 25 of those years in Delphi, C/C++ is never far away. I have an article on the subject that i’ve written for the Idera Community website, so I wont dig too deep into that here — but needless to say, Rad Studio consists of two languages: Object Pascal and C/C++, so no matter how much you love either language, the other is never far away.
So I figured it was time for this old dog to learn some new tricks! I have always said that it’s wise to learn a language immediately below and above your comfort zone. So if Delphi is your favorite language, then C/C++ is below you (meaning: more low level and complex). Above you are languages like JavaScript and C#. Learning JavaScript makes strategic sense (or use DWScript to compile Pascal to JavaScript like I do).
When I started out, the immediate language below Object Pascal was never C, but assembler. So for the longest time I turned to assembler whenever I needed a speed boost; graphics manipulation and processing pixels is especially a field where assembly makes all the difference.
But since C++Builder is indeed an integral part of Rad Studio, and Object Pascal and C/C++ so intimately connected (they have evolved side by side), why not enjoy both assembly and C right?
So I decided to jump back into the saddle and see what I could make of it.
C/C++ is not as hard as you think

I’m having a ball writing C/C++, and just like Delphi – you can start where you are.
While I’m not going to rehash the article I have already prepared for the Idera Community pages here, I do want to encourage people to give it a proper try. I have always said that if you know an archetypal language, you can easily pick up other languages, because the archetypal languages will benefit you for a lifetime. This has to do with archetypal languages operating according to how computers really work; as opposed to optimistic languages (a term from the DB work, optimistic locking), also called contextual languages, like C#, Java, JavaScript etc. are based on how human beings would like things to be.
So I now had a chance to put my money where my mouth is.
When I left C back in the early 90s, I never bothered with OOP. I mean, I used C purely for shared libraries anyways, while the actual programs were done in Pascal or a hybrid language called Blitz Basic. The latter compiled to razor sharp machine code, and you could use inline assembly – which I used a lot back then (very few programmers on those machines went without assembler, it was almost given that you could use 68k in some capacity).
Without ruining the article about to be published, I had a great time with C++Builder. It took a few hours to get my bearings, but since both the VCL and FMX frameworks are there – you can approach C/C++ just like you would Object Pascal. So it’s a matter of getting an overview really.
Needless to say, I’ll be porting a fair share of my libraries to C/C++ when I have time (those that makes sense under that paradigme). It’s always good to push yourself and there are plenty of subtle differences that I found useful.
Quartex Media Desktop
When I last wrote about QTX we were nearing the completion of the FileSystem and Task Management service. The prototype had all its file-handling directly in the core service (or server) which worked just fine — but it was linked to the Smart Pascal RTL. It has taken time to write a new RTL + a full multi-user, platform independent service stack and desktop (phew!) but we are seeing progress!

The QTX Baseline backend services is now largely done
The filesystem service is now largely done! There are a few synchronous calls I want to get rid of, but thankfully my framework has both async and sync variations of all file procedures – so that is now finished.
To make that clearer: first I have to wrap and implement the functionality for the RTL. Once they are in the RTL, I can use those functions to build the service functions. So yeah, it’s been extremely elaborate — but thankfully it’s also become a rich, well organized codebase (both the RTL and the Quartex Media Desktop codebases) – so I think we are ready to get cracking on the core!
The core is still operating with the older API. So our next step is to remove that from the core and instead delegate calls to the filesystem to our new service. So the core will simply be reduced to a post-office or traffic officer if you like. Messages come in from the desktops, and the core delegates the messages to whatever service is in charge of them.
But, this also means that both the core and the desktop must use the new and fancy messages. And this is where I did something very clever.
While I was writing the service, I also write a client class to test (obviously). And the way the core works — means that the same client that the core use to talk to the services — can be used by the desktop as well.
So our work in the desktop to get file-access and drives running again, is to wrap the client in our TQTXDevice ancestor class. The desktop NEVER accesses the API directly. All it knows about are these device drivers (or object instances). Which is how we solve things like DropBox and Google Drive support. The desktop wont have the faintest clue that its using Dropbox, or copying files between a local disk and Google Drive for example — because it only communicates with these device classes.
Recursive stuff
One thing that sucked about node.js function for deleting a folder, is that it’s recursive parameter doesn’t work on Windows or OS X. So I had to implement a full recursive deletefolder routine manually. Not a big thing, but slightly more painful than expected under asynchronous execution. Thankfully, Object Pascal allows for inline defined procedures, so I didn’t have to isolate it in a separate class.
Here is some of the code, a tiny spec compared to the full shabam, but it gives you an idea of what life is like under async conditions:
unit service.file.core; interface {.$DEFINE DEBUG} const CNT_PREFS_DEFAULTPORT = 1883; CNT_PREFS_FILENAME = 'QTXTaskManager.preferences.ini'; CNT_PREFS_DBNAME = 'taskdata.db'; CNT_ZCONFIG_SERVICE_NAME = 'TaskManager'; uses qtx.sysutils, qtx.json, qtx.db, qtx.logfile, qtx.orm, qtx.time, qtx.node.os, qtx.node.sqlite3, qtx.node.zconfig, qtx.node.cluster, qtx.node.core, qtx.node.filesystem, qtx.node.filewalker, qtx.fileapi.core, qtx.network.service, qtx.network.udp, qtx.inifile, qtx.node.inifile, NodeJS.child_process, ragnarok.types, ragnarok.Server, ragnarok.messages.base, ragnarok.messages.factory, ragnarok.messages.network, service.base, service.dispatcher, service.file.messages; type TQTXTaskServiceFactory = class(TMessageFactory) protected procedure RegisterIntrinsic; override; end; TQTXFileWriteCB = procedure (TagValue: variant; Error: Exception); TQTXFileStateCB = procedure (TagValue: variant; Error: Exception); TQTXUnRegisterLocalDeviceCB = procedure (TagValue: variant; DiskName: string; Error: Exception); TQTXRegisterLocalDeviceCB = procedure (TagValue: variant; LocalPath: string; Error: Exception); TQTXFindDeviceCB = procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception); TQTXGetDisksCB = procedure (TagValue: variant; Devices: JDeviceList; Error: Exception); TQTXGetFileInfoCB = procedure (TagValue: variant; LocalName: string; Info: JStats; Error: Exception); TQTXGetTranslatePathCB = procedure (TagValue: variant; Original, Translated: string; Error: Exception); TQTXCheckDevicePathCB = procedure (TagValue: variant; PathName: string; Error: Exception); TQTXServerExecuteCB = procedure (TagValue: variant; Data: string; Error: Exception); TQTXTaskService = class(TRagnarokService) private FPrefs: TQTXIniFile; FLog: TQTXLogEmitter; FDatabase: TSQLite3Database; FZConfig: TQTXZConfigClient; FRegHandle: TQTXDispatchHandle; FRegCount: integer; procedure HandleGetDevices(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleGetDeviceByName(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleCreateLocalDevice(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleDestroyDevice(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleFileRead(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleFileReadPartial(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleGetFileInfo(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleFileDelete(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleFileWrite(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleFileWritePartial(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleFileRename(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleGetDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleMkDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleRmDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure ExecuteExternalJS(Params: array of string; TagValue: variant; const CB: TQTXServerExecuteCB); procedure SendError(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage; Message: string); protected function GetFactory: TMessageFactory; override; procedure SetupPreferences(const CB: TRagnarokServiceCB); procedure SetupLogfile(LogFileName: string;const CB: TRagnarokServiceCB); procedure SetupDatabase(const CB: TRagnarokServiceCB); procedure ValidateLocalDiskName(TagValue: variant; Username, DeviceName: string; CB: TQTXCheckDevicePathCB); procedure RegisterLocalDevice(TagValue: variant; Username, DiskName: string; CB: TQTXRegisterLocalDeviceCB); procedure UnRegisterLocalDevice(TagValue: variant; UserName, DiskName:string; CB: TQTXUnRegisterLocalDeviceCB); procedure GetDevicesForUser(TagValue: variant; UserName: string; CB: TQTXGetDisksCB); procedure FindDeviceByName(TagValue: variant; UserName, DiskName: string; CB: TQTXFindDeviceCB); procedure FindDeviceByType(TagValue: variant; UserName: string; &Type: JDeviceType; CB: TQTXGetDisksCB); procedure GetTranslatedPathFor(TagValue: variant; Username, FullPath: string; CB: TQTXGetTranslatePathCB); procedure GetFileInfo(TagValue: variant; UserName: string; FullPath: string; CB: TQTXGetFileInfoCB); procedure SetupTaskTable(const TagValue: variant; const CB: TRagnarokServiceCB); procedure SetupOperationsTable(const TagValue: variant; const CB: TRagnarokServiceCB); procedure SetupDeviceTable(const TagValue: variant; const CB: TRagnarokServiceCB); procedure AfterServerStarted; override; procedure BeforeServerStopped; override; procedure Dispatch(Socket: TNJWebSocketSocket; Message: TQTXBaseMessage); override; public property Preferences: TQTXIniFile read FPrefs; property Database: TSQLite3Database read FDatabase; procedure SetupService(const CB: TRagnarokServiceCB); constructor Create; override; destructor Destroy; override; end; implementation //############################################################################# // TQTXFileenticationFactory //############################################################################# procedure TQTXTaskServiceFactory.RegisterIntrinsic; begin writeln("Registering task interface"); &Register(TQTXFileGetDeviceListRequest); &Register(TQTXFileGetDeviceByNameRequest); &Register(TQTXFileCreateLocalDeviceRequest); &Register(TQTXFileDestroyDeviceRequest); &Register(TQTXFileReadPartialRequest); &Register(TQTXFileReadRequest); &Register(TQTXFileWritePartialRequest); &Register(TQTXFileWriteRequest); &Register(TQTXFileDeleteRequest); &Register(TQTXFileRenameRequest); &Register(TQTXFileInfoRequest); &Register(TQTXFileDirRequest); &Register(TQTXMkDirRequest); &Register(TQTXRmDirRequest); &Register(TQTXFileRenameRequest); &Register(TQTXFileDirRequest); end; //############################################################################# // TQTXTaskService //############################################################################# constructor TQTXTaskService.Create; begin inherited Create; FPrefs := TQTXIniFile.Create(); FLog := TQTXLogEmitter.Create(); FDatabase := TSQLite3Database.Create(nil); FZConfig := TQTXZConfigClient.Create(); FZConfig.Port := 2292; self.OnUserSignedOff := procedure (Sender: TObject; Username: string) begin WriteToLogF("We got a service signal! User [%s] has signed off completely", [Username]); end; MessageDispatch.RegisterMessage(TQTXFileGetDeviceListRequest, @HandleGetDevices); MessageDispatch.RegisterMessage(TQTXFileGetDeviceByNameRequest, @HandleGetDeviceByName); MessageDispatch.RegisterMessage(TQTXFileCreateLocalDeviceRequest, @HandleCreateLocalDevice); MessageDispatch.RegisterMessage(TQTXFileDestroyDeviceRequest, @HandleDestroyDevice); MessageDispatch.RegisterMessage(TQTXFileReadRequest, @HandleFileRead); MessageDispatch.RegisterMessage(TQTXFileReadPartialRequest, @HandleFileReadPartial); MessageDispatch.RegisterMessage(TQTXFileWriteRequest, @HandleFileWrite); MessageDispatch.RegisterMessage(TQTXFileWritePartialRequest, @HandleFileWritePartial); MessageDispatch.RegisterMessage(TQTXFileInfoRequest, @HandleGetFileInfo); MessageDispatch.RegisterMessage(TQTXFileDeleteRequest, @HandleFileDelete); MessageDispatch.RegisterMessage(TQTXMkDirRequest, @HandleMkDir); MessageDispatch.RegisterMessage(TQTXRmDirRequest, @HandleRmDir); MessageDispatch.RegisterMessage(TQTXFileRenameRequest, @HandleFileRename); MessageDispatch.RegisterMessage(TQTXFileDirRequest, @HandleGetDir); end; destructor TQTXTaskService.Destroy; begin // decouple logger from our instance self.logging := nil; // Release prefs + log FPrefs.free; FLog.free; FZConfig.free; inherited; end; procedure TQTXTaskService.SendError(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage; Message: string); begin var reply := TQTXErrorMessage.Create(request.ticket); try reply.Code := CNT_MESSAGE_CODE_ERROR; reply.Routing.TagValue := Request.Routing.TagValue; reply.Response := Message; if Socket.ReadyState = rsOpen then begin try Socket.Send( reply.Serialize() ); except on e: exception do WriteToLog(e.message); end; end else WriteToLog("Failed to dispatch error, socket is closed error"); finally reply.free; end; end; procedure TQTXTaskService.ExecuteExternalJS(Params: array of string; TagValue: variant; const CB: TQTXServerExecuteCB); begin var LTask: JChildProcess; var lOpts := TVariant.CreateObject(); lOpts.shell := false; lOpts.detached := true; Params.insert(0, '--no-warnings'); // Spawn a new process, this creates a new shell interface try LTask := child_process().spawn('node', Params, lOpts ); except on e: exception do begin if assigned(CB) then CB(TagValue, e.message, e); exit; end; end; // Map general errors on process level LTask.on('error', procedure (error: variant) begin {$IFDEF DEBUG} writeln("error->" + error.toString()); {$ENDIF} WriteToLog(error.toString()); if assigned(CB) then CB(TagValue, "", nil); end); // map stdout so we capture the output LTask.stdout.on('data', procedure (data: variant) begin if assigned(CB) then CB(TagValue, data.toString(), nil); end); // map stderr so we can capture exception messages LTask.stderr.on('data', procedure (error:variant) begin {$IFDEF DEBUG} writeln("stdErr->" + error.toString()); {$ENDIF} if assigned(CB) then CB(TagValue, "", nil); WriteToLog(error.toString()); end); end; function TQTXTaskService.GetFactory: TMessageFactory; begin result := TQTXTaskServiceFactory.Create(); end; procedure TQTXTaskService.SetupService(const CB: TRagnarokServiceCB); begin SetupPreferences( procedure (Error: Exception) begin // No logfile yet setup (!) if Error nil then begin WriteToLog("Preferences setup: Failed!"); WriteToLog(error.message); raise error; end else WriteToLog("Preferences setup: OK"); // logfile-name is always relative to the executable var LLogFileName := TQTXNodeFileUtils.IncludeTrailingPathDelimiter( TQTXNodeFileUtils.GetCurrentDirectory ); LLogFileName += FPrefs.ReadString('log', 'logfile', 'log.txt'); // Port is defined in the ancestor, so we assigns it here Port := FPrefs.ReadInteger('networking', 'port', CNT_PREFS_DEFAULTPORT); SetupLogfile(LLogFileName, procedure (Error: Exception) begin if Error nil then begin WriteToLog("Logfile setup: Failed!"); WriteToLog(error.message); raise error; end else WriteToLog("Logfile setup: OK"); SetupDatabase( procedure (Error: Exception) begin if Error nil then begin WriteToLog("Database setup: Failed!"); WriteToLog(error.message); if assigned(CB) then CB(Error) else raise Error; end else WriteToLog("Database setup: OK"); if assigned(CB) then CB(nil); end); end); end); end; procedure TQTXTaskService.SetupPreferences(const CB: TRagnarokServiceCB); begin var lBasePath := TQTXNodeFileUtils.GetCurrentDirectory; var LPrefsFile := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(LBasePath) + CNT_PREFS_FILENAME; if TQTXNodeFileUtils.FileExists(LPrefsFile) then begin FPrefs.LoadFromFile(nil, LPrefsFile, procedure (TagValue: variant; Error: Exception) begin if Error nil then begin if assigned(CB) then CB(Error) else raise Error; exit; end; if assigned(CB) then CB(nil); end); end else begin var LError := Exception.Create('Could not locate preferences file: ' + LPrefsFile); WriteToLog(LError.message); if assigned(CB) then CB(LError) else raise LError; end; end; procedure TQTXTaskService.SetupLogfile(LogFileName: string;const CB: TRagnarokServiceCB); begin // Attempt to open logfile // Note: Log-object error options is set to throw exceptions try FLog.Open(LogFileName); except on e: exception do begin if assigned(CB) then begin CB(e); exit; end else begin writeln(e.message); raise; end; end; end; // We inherit from TQTXLogObject, which means we can pipe // all errors etc directly using built-in functions. So here // we simply glue our instance to the log-file, and its all good self.Logging := FLog as IQTXLogClient; if assigned(CB) then CB(nil); end; procedure TQTXTaskService.FindDeviceByType(TagValue: variant; UserName: string; &Type: JDeviceType; CB: TQTXGetDisksCB); begin UserName := username.trim().ToLower(); if Username.length < 1 then begin WriteToLog("Failed to lookup disk, username was invalid error"); var lError := EException.Create("Failed to lookup devices, invalid username"); if assigned(CB) then CB(TagValue, nil, lError) else raise lError; exit; end; GetDevicesForUser(TagValue, Username, procedure (TagValue: variant; Devices: JDeviceList; Error: Exception) begin if Error nil then begin if assigned(CB) then CB(TagValue, nil, Error) else raise Error; exit; end; var x := 0; while x < Devices.dlDrives.Count do begin if Devices.dlDrives[x].&Type &Type then begin Devices.dlDrives.delete(x, 1); continue; end; inc(x); end; if assigned(CB) then CB(TagValue, Devices, nil); end); end; procedure TQTXTaskService.FindDeviceByName(TagValue: variant; Username, DiskName: string; CB: TQTXFindDeviceCB); begin UserName := username.trim().ToLower(); if Username.length < 1 then begin var lLogText := "Failed to lookup device, username was invalid error"; WriteToLog(lLogText); var lError := EException.Create(lLogText); if assigned(CB) then CB(TagValue, nil, lError) else raise lError; exit; end; DiskName := DiskName.trim(); if DiskName.length < 1 then begin var lLogText := "Failed to lookup device, disk-name was invalid error"; WriteToLog(lLogText); var lError := EException.Create(lLogText); if assigned(CB) then CB(TagValue, nil, lError) else raise lError; exit; end; GetDevicesForUser(TagValue, Username, procedure (TagValue: variant; Devices: JDeviceList; Error: Exception) begin if Error nil then begin if assigned(CB) then CB(TagValue, nil, Error) else raise Error; exit; end; DiskName := DiskName.trim().ToLower(); var lDiskInfo: JDeviceInfo := nil; for var disk in Devices.dlDrives do begin if disk.Name.ToLower() = DiskName then begin lDiskInfo := disk; break; end; end; if assigned(CB) then CB(TagValue, lDiskInfo, nil); end); end; procedure TQTXTaskService.GetTranslatedPathFor(TagValue: variant; Username, FullPath: string; CB: TQTXGetTranslatePathCB); begin var lParser := TQTXPathParser.Create(); try var lInfo: TQTXPathData; if lparser.Parse(FullPath, lInfo) then begin // Locate the device for the path belonging to the user FindDeviceByName(TagValue, UserName, lInfo.MountPart, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin if Error nil then begin if assigned(CB) then CB(TagValue, FullPath, '', Error) else raise Error; exit; end; if Device.&Type dtLocal then begin var lError := EException.CreateFmt('Failed to translate path, device [%s] is not local error', [Device.Name]); if assigned(CB) then CB(TagValue, FullPath, '', Error) else raise Error; exit; end; // We want the path + filename, so we can append that to // the actual localized filesystem var lExtract := FullPath; delete(lExtract, 1, lInfo.MountPart.Length + 1); // Construct complete storage location var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices'; lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + lExtract; // Return translated path if assigned(CB) then CB(TagValue, FullPath, lFullPath, nil); end); end else begin var lErr := EException.CreateFmt("Invalid path [%s] error", [FullPath]); if assigned(CB) then CB(TagValue, FullPath, '', lErr) else raise lErr; end; finally lParser.free; end; end; procedure TQTXTaskService.GetFileInfo(TagValue: variant; UserName, FullPath: string; CB: TQTXGetFileInfoCB); begin var lParser := TQTXPathParser.Create(); try var lInfo: TQTXPathData; if lparser.Parse(FullPath, lInfo) then begin // Locate the device for the path belonging to the user FindDeviceByName(TagValue, UserName, lInfo.MountPart, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin if Error nil then begin if assigned(CB) then CB(TagValue, '', nil, Error) else raise Error; exit; end; case Device.&Type of dtLocal: begin // We want the path + filename, so we can append that to // the actual localized filesystem var lExtract := FullPath; delete(lExtract, 1, lInfo.MountPart.Length + 1); // Construct complete storage location var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices'; lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + lExtract; // Call the underlying OS to get the file statistics NodeJsFsAPI().lStat(lFullPath, procedure (Error: JError; Stats: JStats) begin if Error nil then begin var lError := EException.Create(Error.message); if assigned(CB) then CB(TagValue, lFullPath, nil, lError) else raise lError; exit; end; // And deliver if assigned(CB) then CB(TagValue, lFullPath, Stats, nil); end); end; dtDropbox, dtGoogle, dtMsDrive: begin var lError := EException.Create("Cloud bindings not activated error"); if assigned(CB) then CB(TagValue, '', nil, lError) end; end; end); end else begin var lErr := EException.CreateFmt("Invalid path [%s] error", [FullPath]); if assigned(CB) then CB(TagValue, '', nil, lErr) else raise lErr; end; finally lParser.free; end; end; procedure TQTXTaskService.GetDevicesForUser(TagValue: variant; Username: string; CB: TQTXGetDisksCB); begin UserName := username.trim().ToLower(); if Username.length < 1 then begin WriteToLog("Failed to lookup devices, username was invalid error"); var lError := EException.Create("Failed to lookup devices, invalid username"); if assigned(CB) then CB(TagValue, nil, lError) else raise lError; exit; end; var lTransaction: TQTXReadTransaction; if not TSQLite3Database(DataBase).CreateReadTransaction(lTransaction) then begin var lErr := EException.Create("Failed to create read-transaction error"); if assigned(cb) then CB(TagValue, nil, lErr) else raise lErr; exit; end; var lQuery := TSQLite3ReadTransaction(lTransaction); lQuery.SQL := "select * from devices where owner=?"; lQuery.Parameters.AddValueOnly(Username); lQuery.Execute( procedure (Sender: TObject; Error: Exception) begin if Error nil then begin if assigned(CB) then CB(TagValue, nil, Error) else raise Error; exit; end; var lDisks := new JDeviceList(); lDisks.dlUser := UserName; for var x := 0 to lQuery.datarows.length-1 do begin var lInfo := new JDeviceInfo(); lInfo.Name := lQuery.datarows[x]["name"]; lInfo.&Type := JDeviceType( lQuery.datarows[x]["type"] ); lInfo.owner := lQuery.datarows[x]["owner"]; lInfo.location := lQuery.datarows[x]["location"]; lInfo.APIKey := lQuery.datarows[x]["apikey"]; lInfo.APISecret := lQuery.datarows[x]["apisecret"]; lInfo.APIPassword := lQuery.datarows[x]["apipassword"]; lInfo.APIUser := lQuery.datarows[x]["apiuser"]; lDisks.dlDrives.add(lInfo); end; try if assigned(CB) then CB(TagValue, lDisks, nil); finally lQuery.free; end; end); end; procedure TQTXTaskService.ValidateLocalDiskName(TagValue: variant; Username, DeviceName: string; CB: TQTXCheckDevicePathCB); begin var Filename := 'disk.' + username + '.' + DeviceName + '.' + ord(JDeviceType.dtLocal).ToString(); var LBasePath := TQTXNodeFileUtils.GetCurrentDirectory(); LBasePath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(LBasePath) + 'userdevices'; // Make sure the device folder is there if not TQTXNodeFileUtils.DirectoryExists(LBasePath) then begin var lError := EException.CreateFmt("Directory not found: %s", [lBasePath]); if assigned(CB) then CB(TagValue, '', lError) else raise lError; exit; end; lBasePath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(LBasePath) + Filename; if TQTXNodeFileUtils.DirectoryExists(LBasePath) then begin var lError := EException.CreateFmt("Path already exist error [%s]", [lBasePath]); if assigned(CB) then CB(TagValue, '', lError) else raise lError; exit; end; // OK, folder is not created yet, so its good to go if assigned(CB) then CB(TagValue, Filename, nil); end; procedure TQTXTaskService.UnRegisterLocalDevice(TagValue: variant; UserName, DiskName: string; CB: TQTXUnRegisterLocalDeviceCB); begin WriteToLogF("Removing local device [%s] for user [%s] ", [DiskName, Username]); // Check username string UserName := username.trim().ToLower(); if Username.length < 1 then begin WriteToLog("Failed to unregister device, username was invalid error"); var lError := EException.Create("Failed to register device, invalid username"); if assigned(CB) then CB(TagValue, DiskName, lError) else raise lError; exit; end; // Check diskname string DiskName := DiskName.trim().ToLower(); if DiskName.length < 1 then begin WriteToLog("Failed to unregister device, disk-name was invalid error"); var lError := EException.Create("Failed to register device, invalid disk-name"); if assigned(CB) then CB(TagValue, DiskName, lError) else raise lError; exit; end; FindDeviceByName(TagValue, Username, DiskName, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin // Did the search fail? if Error nil then begin WriteToLog(Error.message); if assigned(CB) then CB(TagValue, DiskName, Error) else raise Error; exit; end; // Make sure the device is local if Device.&Type dtLocal then begin var lError := EException.CreateFmt('Failed to translate path, device [%s] is not local error', [Device.Name]); if assigned(CB) then CB(TagValue, DiskName, Error) else raise Error; exit; end; // Delete record from database var lWriter: TQTXWriteTransaction; if FDatabase.CreateWriteTransaction(lWriter) then begin lWriter.SQL := "delete from profiles where user = ? and name = ?;"; lWriter.Parameters.AddValueOnly(Username); lWriter.Parameters.AddValueOnly(DiskName); lWriter.Execute( procedure (Sender: TObject; Error: Exception) begin try if Error nil then begin if assigned(CB) then CB(TagValue, DiskName, Error) else raise Error; exit; end; // Construct complete storage location var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices'; lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim(); // Now delete the disk-drive directory TQTXNodeFileUtils.DeleteDirectory(nil, lFullPath, procedure (TagValue: variant; Path: string; Error: Exception) begin if assigned(CB) then CB(TagValue, DiskName, Error) end); finally lWriter.free; lWriter := nil; end; end); end; end); end; procedure TQTXTaskService.RegisterLocalDevice(TagValue: variant; Username, DiskName: string; CB: TQTXRegisterLocalDeviceCB); begin WriteToLogF("Adding local device [%s] for user [%s] ", [DiskName, Username]); UserName := username.trim().ToLower(); if Username.length < 1 then begin WriteToLog("Failed to register device, username was invalid error"); var lError := EException.Create("Failed to register device, invalid username"); if assigned(CB) then CB(TagValue, '', lError) else raise lError; exit; end; DiskName := DiskName.trim().ToLower(); if DiskName.length < 1 then begin WriteToLog("Failed to register device, disk-name was invalid error"); var lError := EException.Create("Failed to register device, invalid disk-name"); if assigned(CB) then CB(TagValue, '', lError) else raise lError; exit; end; FindDeviceByName(TagValue, Username, DiskName, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin // Did the search fail? if Error nil then begin WriteToLog(Error.message); if assigned(CB) then CB(TagValue, '', Error) else raise Error; exit; end; // Does a device that match already exist? if Device nil then begin var lError := EException.CreateFmt("Failed to create device [%s], device already exists", [DiskName]); if assigned(CB) then CB(TagValue, '', lError) else raise lError; exit; end; // make sure the device-folder does not exist, so we can create it ValidateLocalDiskName(TagValue, Username, DiskName, procedure (TagValue: variant; PathName: string; Error: Exception) begin if Error nil then begin if assigned(CB) then CB(TagValue, '', Error) else raise Error; exit; end; // ValidateLocalDiskName only returns the valid directory-name, // not a full path -- so we need to build up the full targetpath var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices'; lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + PathName; TQTXNodeFileUtils.CreateDirectory(nil, lFullPath, procedure (TagValue: variant; Path: string; Error: exception) begin if Error nil then begin var lError := EException.CreateFmt("Failed to create device [%s] with path: %s", [DiskName, lFullPath]); if assigned(CB) then CB(TagValue, PathName, lError) else raise lError; exit; end; FDatabase.Execute( #'insert into devices (type, owner, name, location) values(?, ?, ?, ?);', [ord(JDeviceType.dtLocal), UserName, Diskname, PathName] , procedure (Sender: TObject; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); if assigned(CB) then CB(TagValue, PathName, Error) else raise Error; exit; end; WriteToLogF("Device [%s] added to database user [%s]", [DiskName, UserName]); if assigned(CB) then CB(TagValue, PathName, nil); end); end); end); end); end; procedure TQTXTaskService.SetupDeviceTable(const TagValue: variant; const CB: TRagnarokServiceCB); begin FDatabase.Execute( #' create table if not exists devices ( id integer primary key AUTOINCREMENT, type integer, owner text, name text, location text, apikey text, apisecret text, apipassword text, apiuser text ); ', [], procedure (Sender: TObject; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); if assigned(CB) then CB(Error) else raise Error; exit; end else if assigned(CB) then CB(nil); end); end; procedure TQTXTaskService.SetupTaskTable(const TagValue: variant; const CB: TRagnarokServiceCB); begin FDatabase.Execute( #' create table if not exists tasks ( id integer primary key AUTOINCREMENT, state integer, username text, created real ); ', [], procedure (Sender: TObject; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); if assigned(CB) then CB(Error) else raise Error; exit; end else if assigned(CB) then CB(nil); end); end; procedure TQTXTaskService.SetupOperationsTable(const TagValue: variant; const CB: TRagnarokServiceCB); begin FDatabase.Execute( #' create table if not exists operations ( id integer primary key AUTOINCREMENT, username text, taskid integer, name text, filename text ); ', [], procedure (Sender: TObject; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); if assigned(CB) then CB(Error) else raise Error; exit; end else if assigned(CB) then CB(nil); end); end; procedure TQTXTaskService.SetupDatabase(const CB: TRagnarokServiceCB); begin // Try to read database-path from preferences file var LDbFileToOpen := FPrefs.ReadString("database", "database_name", ""); // Trim away spaces, check if there is a filename LDbFileToOpen := LDbFileToOpen.trim(); if LDbFileToOpen.length < 1 then begin // No filename? Fall back on pre-defined file in CWD var LBasePath := TQTXNodeFileUtils.GetCurrentDirectory(); LDbFileToOpen := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(LBasePath) + CNT_PREFS_DBNAME; end; FDatabase.AccessMode := TSQLite3AccessMode.sqaReadWriteCreate; FDatabase.Open(LDbFileToOpen, procedure (Sender: TObject; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); if assigned(CB) then CB(Error) else raise Error; exit; end; WriteToLog("Initializing task table"); SetupTaskTable(nil, procedure (Error: exception) begin if Error nil then begin WriteToLog("Tasks initialized: **failed"); WriteToLog(error.message); if assigned(CB) then CB(Error) else raise Error; exit; end else writeToLog("Tasks initialized: OK"); WriteToLog("Initializing operations table"); SetupOperationsTable(nil, procedure (Error: exception) begin if Error nil then begin WriteToLog("Operations initialized: **failed"); WriteToLog(error.message); if assigned(CB) then CB(Error); exit; end else writeToLog("Operations initialized: OK"); WriteToLog("Initializing device table"); SetupDeviceTable(nil, procedure (Error: exception) begin if Error nil then begin WriteToLog("Device-table initialized: **failed"); WriteToLog(error.message); if assigned(CB) then CB(Error); exit; end else writeToLog("Device-table initialized: OK"); if assigned(CB) then CB(nil); end); end); end); end); end; procedure TQTXTaskService.HandleFileRead(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileReadRequest(request); var lUserName := lRequest.UserName; var lFileName := lRequest.FileName; // Check filename length if lFileName.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; GetFileInfo(lRequest, lUserName, lFileName, procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; var lOptions: TReadFileOptions; lOptions.encoding := 'binary'; NodeJsFsAPI().readFile(LocalFile, lOptions, procedure (Error: JError; Data: JNodeBuffer) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; var lResponse := TQTXFileReadResponse.Create(Request.Ticket); lResponse.UserName := lUserName; lResponse.Routing.TagValue := request.routing.tagValue; lResponse.FileName := lFileName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; // Convert filedata in one pass try var lConvert := TDataTypeConverter.Create(); try lResponse.Attachment.AppendBytes( lConvert.TypedArrayToBytes(Data) ); finally lConvert.free; end; except on e: exception do begin WriteToLog(e.message); SendError(Socket, Request, e.Message); exit; end; end; try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end); end; procedure TQTXTaskService.HandleFileReadPartial(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileReadPartialRequest(request); var lUserName := lRequest.UserName; var lFileName := lRequest.FileName; var lStart := lRequest.Offset; var lSize := lRequest.Size; // Check filename length if lFileName.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; if lSize < 1 then begin SendError(Socket, Request, "Read failed, invalid size error"); exit; end; if lStart < 0 then begin SendError(Socket, Request, "Read failed, invalid offset error"); exit; end; GetFileInfo(lRequest, lUserName, lFileName, procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; if lStart > Info.size then begin SendError(Socket, Request, "Read failed, offset beyond filesize error"); exit; end; NodeJsFsAPI().open(LocalFile, "r", procedure (Error: JError; Fd: THandle) begin if error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; var Data = new JNodeBuffer(lSize); NodeJsFsAPI().read(Fd, Data, 0, lSize, lStart, procedure (Error: JError; BytesRead: integer; buffer: JNodeBuffer) begin if Error nil then begin NodeJsFsAPI().closeSync(Fd); WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; // Close the file-handle and return data NodeJsFsAPI().close(Fd, procedure (Error: JError) begin var lResponse := TQTXFileReadPartialResponse.Create(Request.Ticket); lResponse.UserName := lUserName; lResponse.Routing.TagValue := request.routing.tagValue; lResponse.FileName := lFileName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; // Only encode data if read if BytesRead > 0 then begin // Convert filedata in one pass try var lConvert := TDataTypeConverter.Create(); try lResponse.Attachment.AppendBytes( lConvert.TypedArrayToBytes(buffer) ); finally lConvert.free; end; except on e: exception do begin WriteToLog(e.message); SendError(Socket, Request, e.Message); exit; end; end; end; try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end); end); end); end; procedure TQTXTaskService.HandleFileWrite(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileWriteRequest(request); var lFileName := lRequest.FileName.trim(); var lUserName := lRequest.UserName.trim(); var FullPath := lFileName; // Check filename length if lFileName.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; var lParser := TQTXPathParser.Create(); try var lInfo: TQTXPathData; if lparser.Parse(FullPath, lInfo) then begin // Locate the device for the path belonging to the user FindDeviceByName(nil, lUserName, lInfo.MountPart, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; case Device.&Type of dtLocal: begin // We want the path + filename, so we can append that to // the actual localized filesystem var lExtract := FullPath; delete(lExtract, 1, lInfo.MountPart.Length + 1); // Construct complete storage location var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices'; lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + lExtract; // Extract data to be appended, if any // note: null bytes should be allowed, it should just create the file var lBytes: array of UInt8; if lRequest.attachment.Size > 0 then lBytes := lRequest.Attachment.ToBytes(); // Write the data to the file NodeJsFsAPI().writeFile(lFullPath, lBytes, procedure (Error: JError) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; // Setup response object var lResponse := TQTXFileWriteResponse.Create(lRequest.Ticket); lResponse.UserName := lUserName; lResponse.FileName := lFileName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; // Send success response try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end; dtDropbox, dtGoogle, dtMsDrive: begin var lErrorText := Format("Clound bindings not active error [%s]", [lRequest.FileName]); WriteToLog(lErrorText); SendError(Socket, Request, lErrorText); end; end; end); end else begin SendError(Socket, Request, format("Invalid path [%s] error", [FullPath])); end; finally lParser.free; end; end; procedure TQTXTaskService.HandleFileWritePartial(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileWritePartialRequest(request); var lFileName := lRequest.FileName.trim(); var lUserName := lRequest.UserName.trim(); var lFileOffset := lRequest.Offset; // Check filename length if lFileName.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; var FullPath := lFileName; var lParser := TQTXPathParser.Create(); try var lInfo: TQTXPathData; if lparser.Parse(FullPath, lInfo) then begin // Locate the device for the path belonging to the user FindDeviceByName(nil, lUserName, lInfo.MountPart, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; case Device.&Type of dtLocal: begin // We want the path + filename, so we can append that to // the actual localized filesystem var lExtract := FullPath; delete(lExtract, 1, lInfo.MountPart.Length + 1); // Construct complete storage location var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices'; lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + lExtract; // Extract data to be appended, if any // note: null bytes should be allowed, it should just create the file var lBytes: array of UInt8; if lRequest.attachment.Size > 0 then lBytes := lRequest.Attachment.ToBytes(); var lAccess := TQTXNodeFile.Create(); lAccess.Open(lFullPath, TQTXNodeFileMode.nfWrite, procedure (Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; lAccess.Write(lBytes, lFileOffset, procedure (Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; // Setup response object var lResponse := TQTXFileWriteResponse.Create(lRequest.Ticket); lResponse.UserName := lUserName; lResponse.FileName := lFileName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; // Send success response try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end); end; dtDropbox, dtGoogle, dtMsDrive: begin var lErrorText := Format("Clound bindings not active error [%s]", [lRequest.FileName]); WriteToLog(lErrorText); SendError(Socket, Request, lErrorText); end; end; end); end else begin SendError(Socket, Request, format("Invalid path [%s] error", [FullPath])); end; finally lParser.free; end; end; procedure TQTXTaskService.HandleRmDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXRmDirRequest(request); var lUserName := lRequest.UserName.trim(); var lDirPath := lRequest.DirPath.trim(); if lDirPath.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lDirPath) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; var lParser := TQTXPathParser.Create(); try var lInfo: TQTXPathData; if lParser.Parse(lDirPath, lInfo) then begin GetTranslatedPathFor(nil, lUserName, lDirPath, procedure (TagValue: variant; Original, Translated: string; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; if not TQTXNodeFileUtils.DirectoryExists(Translated) then begin WriteToLogF("RmDir Failed, directory [%s] does not exist", [Translated]); SendError(Socket, Request, Format("RmDir failed, directory [%s] does not exist", [Original])); exit; end; TQTXNodeFileUtils.DeleteDirectory(nil, Translated, procedure (TagValue: variant; Path: string; Error: Exception) begin if error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; // Setup response object var lResponse := TQTXRmDirResponse.Create(lRequest.Ticket); lResponse.UserName := lUserName; lResponse.DirPath := lDirPath; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; lResponse.Routing.TagValue := lRequest.Routing.TagValue; // Send success response try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end); end else begin var lText := format("RmDir failed, invalid path [%s] error", [lDirPath]); WriteToLog(lText); SendError(Socket, Request, lText); end; finally lParser.free; end; end; procedure TQTXTaskService.HandleMkDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXMkDirRequest(request); var lUserName := lRequest.UserName.trim(); var lDirPath := lRequest.DirPath.trim(); if lDirPath.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lDirPath) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; var lParser := TQTXPathParser.Create(); try var lInfo: TQTXPathData; if lparser.Parse(lDirPath, lInfo) then begin GetTranslatedPathFor(nil, lUserName, lDirPath, procedure (TagValue: variant; Original, Translated: string; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; TQTXNodeFileUtils.DirectoryExists(nil, Translated, procedure (TagValue: variant; Path: string; Error: Exception) begin if Error nil then begin WriteToLogF("MkDir Failed, directory [%s] already exists", [Translated]); SendError(Socket, Request, Format("MkDir Failed, directory [%s] already exists", [Original])); exit; end; TQTXNodeFileUtils.CreateDirectory(nil, Translated, procedure (TagValue: variant; Path: string; Error: Exception) begin if Error nil then begin WriteToLogF("MkDir Failed, directory [%s] could not be created", [Original]); SendError(Socket, Request, Format("MkDir Failed, directory [%s] could not be created", [Translated])); exit; end; // Setup response object var lResponse := TQTXMkDirResponse.Create(lRequest.Ticket); lResponse.UserName := lUserName; lResponse.DirPath := lDirPath; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; lResponse.Routing.TagValue := lRequest.Routing.TagValue; // Send success response try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end); end); end else begin var lText := format("MkDir Failed, invalid path [%s] error", [lDirPath]); WriteToLog(lText); SendError(Socket, Request, lText); end; finally lParser.free; end; end; procedure TQTXTaskService.HandleFileDelete(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileDeleteRequest(Request); var lUserName := lRequest.UserName.trim(); var lFileName := lRequest.FileName.trim(); if lFileName.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; GetFileInfo(lRequest, lUserName, lFileName, procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; if not Info.isFile then begin SendError(Socket, Request, "Filesystem object is not a file error"); exit; end; NodeJsFsAPI().unlink(LocalFile, procedure (Error: JError) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.message); exit; end; var lResponse := new TQTXFileDeleteResponse(lRequest.Ticket); lResponse.Routing.TagValue := request.Routing.TagValue; lResponse.UserName := lUserName; lResponse.FileName := lFileName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end); end; procedure TQTXTaskService.HandleFileRename(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileRenameRequest(Request); var lUserName := lRequest.UserName.trim(); var lFileName := lRequest.FileName.trim(); var lNewName := lRequest.NewName.trim(); // Check filename length if lFileName.length < 1 then begin SendError(Socket, Request, Format("Invalid or empty from-filename [%s] error", [lFileName]) ); exit; end; // check newname length if lNewName.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; if pos(lTemp, lNewName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; if pos(lTemp, lNewName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; GetFileInfo(lRequest, lUserName, lFileName, procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; if not Info.isFile then begin SendError(Socket, Request, "Filesystem object is not a file error"); exit; end; GetTranslatedPathFor(nil, lUsername, lNewName, procedure (TagValue: variant; Original, Translated: string; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; NodeJsFsAPI().rename(LocalFile, Translated, procedure (Error: JError) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.message); exit; end; var lResponse := new TQTXFileRenameResponse(lRequest.Ticket); lResponse.Routing.TagValue := request.Routing.TagValue; lResponse.UserName := lUserName; lResponse.FileName := lFileName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end); end); end; procedure TQTXTaskService.HandleGetDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileDirRequest(Request); var lUserName := lRequest.UserName.trim(); var lPath := lRequest.Path.trim(); // prevent path escape attempts var lTemp := "../"; if pos(lTemp, lPath) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lPath) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; GetTranslatedPathFor(nil, lUserName, lPath, procedure (TagValue: variant; Original, Translated: string; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; //writeln("Translated path is:" + Translated); if not TQTXNodeFileUtils.DirectoryExists(Translated) then begin WriteToLogF("GetDir Failed, directory [%s] does not exist", [Translated]); SendError(Socket, Request, Format("GetDir failed, directory [%s] does not exist", [Original])); exit; end; var lWalker := TQTXFileWalker.Create(); lWalker.Examine(Translated, procedure (Sender: TQTXFileWalker; Error: EException) begin if Error nil then begin WriteToLogF("GetDir Failed: %s", [Error.Message]); SendError(Socket, Request, Format("GetDir failed: %s", [Error.Message])); exit; end; // Get the directory data, swap out the path // record with the original [amiga] style path var lData := Sender.ExtractList(); lData.dlPath := Original; var lResponse := new TQTXFileDirResponse(lRequest.Ticket); lResponse.Routing.TagValue := request.Routing.TagValue; lResponse.UserName := lUserName; lResponse.Path := lPath; lResponse.Assign( lData ); try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; // release instance in 100ms TQTXDispatch.execute(procedure () begin try lWalker.free except on e: exception do begin WriteToLogF("Failed to release file-walker instance: %s", [e.message]); end; end; end, 100); end); end); end; procedure TQTXTaskService.HandleGetFileInfo(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileInfoRequest(Request); var lUserName := lRequest.UserName.trim(); var lFileName := lRequest.FileName.trim(); // prevent path escape attempts var lTemp := "../"; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; GetFileInfo(lRequest, lUserName, lFileName, procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; // Collect the data var lData := new JFileItem(); lData.diFileName := lFileName; lData.diFileType := if Info.isFile then JFileItemType.wtFile else JFileItemType.wtFolder; lData.diFileSize := Info.size; lData.diFileMode := IntToStr(Info.mode); lData.diCreated := TDateUtils.FromJsDate( Info.cTime ); lData.diModified := TDateUtils.FromJsDate( Info.mTime ); var lResponse := new TQTXFileInfoResponse(lRequest.Ticket); lResponse.Routing.TagValue := request.Routing.TagValue; lResponse.UserName := lUserName; lResponse.FileName := lFileName; lResponse.Assign(lData); try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end; procedure TQTXTaskService.HandleDestroyDevice(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lMessage := TQTXFileDestroyDeviceRequest(request); // This will also destroy any files + unregister the device in the // database table for the service -- do not mess with this! UnRegisterLocalDevice(nil, lMessage.Username, lMessage.DeviceName, procedure (TagValue: variant; LocalPath: string; Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; var lResponse := TQTXFileDestroyDeviceResponse.Create(request.ticket); lResponse.UserName := lMessage.UserName; lResponse.DeviceName := lMessage.DeviceName; lResponse.Routing.TagValue := Request.Routing.TagValue; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; try Socket.Send( lResponse.Serialize() ); except on e: exception do begin WriteToLog(e.message); end; end; end); end; procedure TQTXTaskService.HandleCreateLocalDevice(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lMessage := TQTXFileCreateLocalDeviceRequest(request); // Attempt to register. // NOTE: This will automatically create a matching folder // under $cwd/userdevices/[calculated_name_of_device] RegisterLocalDevice(nil, lMessage.Username, lMessage.DeviceName, procedure (TagValue: variant; LocalPath: string; Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; FindDeviceByName(nil, lMessage.Username, lMessage.DeviceName, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; var lResponse := TQTXFileCreateLocalDeviceResponse.Create(request.ticket); lResponse.UserName := lMessage.UserName; lResponse.Routing.TagValue := Request.Routing.TagValue; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; if Device nil then lResponse.assign(Device); try Socket.Send( lResponse.Serialize() ); except on e: exception do begin WriteToLog(e.message); end; end; end); end); end; procedure TQTXTaskService.HandleGetDeviceByName(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lMessage := TQTXFileGetDeviceByNameRequest(request); FindDeviceByName(nil, lMessage.Username, lMessage.DeviceName, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; var lResponse := TQTXFileGetDeviceByNameResponse.Create(request.ticket); lResponse.UserName := lMessage.UserName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; if Device nil then lResponse.assign(Device); try Socket.Send( lResponse.Serialize() ); except on e: exception do begin WriteToLog(e.message); end; end; end); end; procedure TQTXTaskService.HandleGetDevices(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lMessage := TQTXFileGetDeviceListRequest(Request); GetDevicesForUser(nil, lMessage.Username, procedure (TagValue: variant; Devices: JDeviceList; Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; var lResponse := TQTXFileGetDeviceListResponse.Create(request.ticket); lResponse.UserName := lMessage.UserName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; if Devices nil then lResponse.assign(Devices); try Socket.Send( lResponse.Serialize() ); except on e: exception do begin WriteToLog(e.message); end; end; end); end; procedure TQTXTaskService.AfterServerStarted; begin inherited; // Check prefs if zconfig should be applied if self.FPrefs.ReadBoolean("zconfig", "active", false) then begin // ZConfig should only run on the master instance. // We dont want to register our endpoint for each worker if NodeJSClusterAPI().isWorker then exit; writeln("Setting up Zero-Configuration layer"); FZConfig.port := FPrefs.ReadInteger('zconfig', 'bindport', 2109); FZConfig.address := GetMachineIP(); FZConfig.Start(nil, procedure (Sender: TObject; TagValue: variant; Error: Exception) begin if FPrefs.ReadBoolean("zconfig", "broadcast", true) then FZConfig.Socket.setBroadcast(true); // Build up the endpoint (URL) for our websocket server var lEndpoint := ''; if FPrefs.ReadBoolean('networking', 'secure', false) then lEndpoint := 'wss://' else lEndpoint := 'ws://'; lEndpoint += GetMachineIP(); lEndpoint += ':' + Port.ToString(); // Ping the ZConfig service on interval, until our service is registered // We keep track of the interval handle so we can stop calling on interval later FRegHandle := TQTXDispatch.SetInterval( procedure () begin inc(FRegCount); // Only output once to avoid overkill in the log if FRegCount = 1 then WriteToLogF("ZConfig registration begins [%s]", [lEndpoint]); FZConfig.RegisterService(nil, CNT_ZCONFIG_SERVICE_NAME, SERVICE_ID_TASKMANAGER, lEndpoint, procedure (TagValue: variant; Error: Exception) begin if Error = nil then begin WriteToLog("Service registered"); TQTXDispatch.ClearInterval(FRegHandle); FRegCount := 0; exit; end; end); end, 1000); end); end; end; procedure TQTXTaskService.BeforeServerStopped; begin inherited; end; procedure TQTXTaskService.Dispatch(Socket: TNJWebSocketSocket; Message: TQTXBaseMessage); begin var LInfo := MessageDispatch.GetMessageInfoForClass(Message); if LInfo nil then begin try LInfo.MessageHandler(Socket, Message); except on e: exception do begin //Log error WriteToLog(e.message); end; end; end; end; end.
Hydra, what’s the big deal anyway?
RemObjects Hydra is a product I have used for years in concert with Delphi, and like most developers that come into contact with RemObjects products – once the full scope of the components hit you, you never want to go back to not using Hydra in your applications.
Note: It’s easy to dismiss Hydra as a “Delphi product”, but Hydra for .Net and Java does the exact same thing, namely let you mix and match modules from different languages in your programs. So if you are a C# developer looking for ways to incorporate Java, Delphi, Elements or Freepascal components in your application, then keep reading.
But let’s start with what Hydra can do for Delphi developers.
What is Hydra anyways?
Hydra is a component package for Delphi, Freepascal, .Net and Java that takes plugins to a whole new level. Now bear with me for a second, because these plugins is in a completely different league from anything you have used in the past.
In short, Hydra allows you to wrap code and components from other languages, and use them from Delphi or Lazarus. There are thousands of really amazing components for the .Net and Java platforms, and Hydra allows you compile those into modules (or “plugins” if you prefer that); modules that can then be used in your applications like they were native components.

Hydra, here using a C# component in a Delphi application
But it doesn’t stop there; you can also mix VCL and FMX modules in the same application. This is extremely powerful since it offers a clear path to modernizing your codebase gradually rather than doing a time consuming and costly re-write.
So if you want to move your aging VCL codebase to Firemonkey, but the cost of having to re-write all your forms and business logic for FMX would break your budget -that’s where Hydra gives you a second option: namely that you can continue to use your VCL code from FMX and refactor the application in your own tempo and with minimal financial impact.
The best of all worlds
Not long ago RemObjects added support for Lazarus (Freepascal) to the mix, which once again opens a whole new ecosystem that Delphi, C# and Java developers can benefit from. Delphi has a lot of really cool components, but Lazarus have components that are not always available for Delphi. There are some really good developers in the Freepascal community, and you will find hundreds of components and classes (if not thousands) that are open-source; For example, Lazarus has a branch of Synedit that is much more evolved and polished than the fork available for Delphi. And with Hydra you can compile that into a module / plugin and use it in your Delphi applications.
This is also true for Java and C# developers. Some of the components available for native languages might not have similar functionality in the .Net world, and by using Hydra you can tap into the wealth that native languages have to offer.
As a Delphi or Freepascal developer, perhaps you have seen some of the fancy grids C# and Java coders enjoy? Developer Express have some of the coolest components available for any platform, but their focus is more on .Net these days than Delphi. They do maintain the control packages they have, but compared to the amount of development done for C# their Delphi offerings are abysmal. So with Hydra you can tap into the .Net side of things and use the latest components and libraries in your Delphi applications.
Financial savings
One of coolest features of Hydra, is that you can use it across Delphi versions. This has helped me leverage the price-tag of updating to the latest Delphi.
It’s easy to forget that whenever you update Delphi, you also need to update the components you have bought. This was one of the reasons I was reluctant to upgrade my Delphi license until Embarcadero released Delphi 10.2. Because I had thousands of dollars invested in components – and updating all my licenses would cost a small fortune.
So to get around this, I put the components into a Hydra module and compiled that using my older Delphi. And then i simply used those modules from my new Delphi installation. This way I was able to cut cost by thousands of dollars and enjoy the latest Delphi.

Using Firemonkey controls under VCL is easy with Hydra
A couple of years back I also took the time to wrap a ton of older components that work fine but are no longer maintained or sold. I used an older version of Delphi to get these components into a Hydra module – and I can now use those with Delphi 10.3 (!). In my case there was a component-set for working closely with Active Directory that I have used in a customer’s project (and much faster than having to go the route via SQL). The company that made these don’t exist any more, and I have no source-code for the components.
The only way I could have used these without Hydra, would be to compile them into a .dll file and painstakingly export every single method (or use COM+ to cross the 32-bit / 64-bit barrier), which would have taken me a week since we are talking a large body of quality code. With Hydra i was able to wrap the whole thing in less than an hour.
I’m not advocating that people stop updating their components. But I am very thankful for the opportunity to delay having to update my entire component stack just to enjoy a modern version of Delphi.
Hydra gives me that opportunity, which means I can upgrade when my wallet allows it.
Building better applications
There is also another side to Hydra, namely that it allows you to design applications in a modular way. If you have the luxury of starting a brand new project and use Hydra from day one, you can isolate each part of your application as a module. Avoiding the trap of monolithic applications.

Hydra for .Net allows you to use Delphi, Java and FPC modules under C#
This way of working has great impact on how you maintain your software, and consequently how you issue hotfixes and updates. If you have isolated each key part of your application as separate modules, you don’t need to ship a full build every time.
This also safeguards you from having all your eggs in one basket. If you have isolated each form (for example) as separate modules, there is nothing stopping you from rewriting some of these forms in another language – or cross the VCL and FMX barrier. You have to admit that being able to use the latest components from Developer Express is pretty cool. There is not a shadow of a doubt that Developer-Express makes the best damn components around for any platform. There are many grids for Delphi, but they cant hold a candle to the latest and greatest from Developer Express.
Why can’t I just use packages?
If you are thinking “hey, this sounds exactly like packages, why should I buy Hydra when packages does the exact same thing?“. Actually that’s not how packages work for Delphi.
Delphi packages are cool, but they are also severely limited. One of the reasons you have to update your components whenever you buy a newer version of Delphi, is because packages are not backwards compatible.

Delphi packages are great, but severely limited
A Delphi package must be compiled with the same RTL as the host (your program), and version information and RTTI must match. This is because packages use the same RTL and more importantly, the same memory manager.
Hydra modules are not packages. They are clean and lean library files (*.dll files) that includes whatever RTL you compiled them with. In other words, you can safely load a Hydra module compiled with Delphi 7, into a Delphi 10.3 application without having to re-compile.
Once you start to work with Hydra, you gradually build up modules of functionality that you can recycle in the future. In many ways Hydra is a whole new take on components and RAD. This is how Delphi packages and libraries should have been.
Without saying anything bad about Delphi, because Delphi is a system that I love very much; but having to update your entire component stack just to use the latest Delphi, is sadly one of the factors that have led developers to abandon the platform. If you have USD 10.000 in dependencies, having to pay that as well as buying Delphi can be difficult to justify; especially when comparing with other languages and ecosystems.
For me, Hydra has been a tremendous boon for Delphi. It has allowed me to keep current with Delphi and all it’s many new features, without losing the money I have already invested in components packages.
If you are looking for something to bring your product to the next level, then I urge you to spend a few hours with Hydra. The documentation is exceptional, the features and benefits are outstanding — and you will wonder how you ever managed to work without them.
External resources
Disclaimer: I am not a salesman by any stretch of the imagination. I realize that promoting a product made by the company you work for might come across as a sales pitch; but that’s just it: I started to work for RemObjects for a reason. And that reason is that I have used their products since they came on the market. I have worked with these components long before I started working at RemObjects.
.NetRocks, you made my day!
A popular website for .Net developers is called dot-net-rocks. This is an interesting site that has been going for a while now; well worth the visit if you do work with the .Net framework via RemObjects Elements, VS or Mono.
Now it turns out that the guys over at dot–net-rocks just did an episode on their podcast where they open by labeling me as a “raving lunatic” (I clearly have my moments); which I find absolutely hilarious, but not for the same reasons as them.
Long story short: They are doing a podcast on how to migrate legacy Delphi applications to C#, and in that context they somehow tracked down an article I posted way back in 2016, which was meant as a satire piece. Now don’t get me wrong, there are serious points in the article, like how the .Net framework was modeled on the Delphi VCL, and how the concepts around CLR and JIT were researched at Borland; but the tone of the whole thing, the “larger than life” claims etc. was meant to demonstrate just how some .Net developers behave when faced with alternative eco-systems. Having managed some 16+ usergroups for Delphi, C#, JavaScript (a total of six languages) on Facebook for close to 15 years, as well as working for Embarcadero that makes Delphi -I speak from experience.
It might be news to these guys that large companies around Europe is still using Delphi, modern Delphi, and that Object Pascal as a language scores well on the Tiobi index of popular programming languages. And no amount of echo-chamber mentality is going to change that fact. Heck, as late as 2018 and The Walt Disney Company wanted to replace C# with Delphi, because it turns out that bytecodes and embedded tech is not the best combination (cpu spikes when the GC kicks in, no real-time interrupt handling possible, GPIO delays, the list goes on).
I mean, the post i made back in 2016 is such obvious, low-hanging fruit for a show their size to pound on. You have this massive show that takes on a single, albeit ranting (and probably a bit of a lunatic if I don’t get my coffee) coder’s post. Underlying in the process how little they know about the object pascal community at large. They just demonstrated my point in bold, italic and underline 😀
Look before you shoot
DotNetRocks is either oblivious that Delphi still have millions of users around the world, or that Pascal is in fact available for .Net (which is a bit worrying since .Net is supposed to be their game). The alternative is that the facts I listed hit a little too close to home. I’ll leave it up to the reader to decide. Microsoft has lost at least 10 Universities around Europe to Delphi in 2018 that I know of, two of them Norwegian where I was personally involved in the license sales. While only speculation, I do find the timing for their podcast and focus on me in particular to be, “curious”.
And for the record, the most obvious solution when faced with “that legacy Delphi project”, is to just go and buy a modern version of Delphi. DotNetRocks delivered a perfect example of that very arrogance my 2016 post was designed to convey; namely that “brogrammers” often act like Delphi 7 was the last Delphi. They also resorted to lies to sell their points: I never said that Anders was dogged for creating Delphi. Quite the opposite. I simply underlined that by ridiculing Delphi in one hand, and praising it’s author with the other – you are indirectly (and paradoxically) invalidating half his career. Anders is an awesome developer, but why exclude how he evolved his skills? Ofcourse Ander’s products will have his architectural signature on them.
Not once did they mention Embarcadero or the fact that Delphi has been aggressively developed since Borland kicked the bucket. Probably hoping that undermining the messenger will somehow invalidate the message.

Porting Delphi to C# manually? Ok.. why not install Elements and just compile it into an assembly? You don’t even have to leave Visual Studio
Also, such an odd podcast for professional developers to run with. I mean, who the hell converts a Delphi project to C# manually? It’s like listening to a graphics artist that dont know that Photoshop and Illustrator are the de-facto tools to use. How is that even possible? A website dedicated to .Net, yet with no insight into the languages that run on the CLR? Wow.
If you want to port something from Delphi to .Net, you don’t sit down and manually convert stuff. You use proper tools like Elements from RemObjects; This gives you Object-Pascal for .Net (so a lot of code will compile just fine with only minor changes). Elements also ships with source-conversion tools, so once you have it running under Oxygene Pascal (the dialect is called Oxygene) you either just use the assemblies — or convert the Pascal code to C# through a tool called an Oxidizer.

The most obvious solution is to just upgrade to a Delphi version from this century
The other solution is to use Hydra, also a RemObjects product. They can then compile the Delphi code into a library (including visual parts like forms and frames), and simply use that as any other assembly from within C#. This allows you to gradually phase out older parts without breaking the product. You can also use C# assemblies from Delphi with Hydra.
So by all means, call me what you like. You have only proved my point so far. You clearly have zero insight into the predominant Object-Pascal eco-systems, you clearly don’t know the tools developers use to interop between arcetypical and contextual languages — and instead of fact checking some of the points I made, dry humor notwithstanding, you just reacted like brogrammers do.
Well, It’s been weeks since I laughed this hard 😀 You really need to check before you pick someone to verbally abuse on the first date, because you might just bite yourself in the arse here he he
Cheers
Check out RemObjects Remoting SDK
RemObjects Remoting SDK is one of those component packages that have become more than the sum of it’s part. Just like project Jedi has become standard equipment almost, Remoting SDK is a system that all Delphi and Freepascal developers should have in their toolbox.
In this article I’m going to present the SDK in broad strokes; from a viewpoint of someone who haven’t used the SDK before. There are still a large number of Delphi developers that don’t know it even exists – hopefully this post will shed some light on why the system is worth every penny and what it can do for you.
I should also add, that this is a personal blog. This is not an official RemObjects presentation, but a piece written by me based on my subjective experience and notions. We have a lot of running dialog at Delphi Developer on Facebook, so if I read overly harsh on a subject, that is my personal view as a Delphi Developer.
Stop re-inventing the wheel
Delphi has always been a great tool for writing system services. It has accumulated a vast ecosystem of non-visual components over the years, both commercial and non-commercial, and this allows developers to quickly aggregate and expose complex behavior — everything from graphics processing to databases, file processing to networking.
The challenge for Delphi is that writing large composite systems, where you have more than a single service doing work in concert, is not factored into the RTL or project type. Delphi provides a bare-bone project type for system services, and that’s it. Depending on how you look at it, it’s either a blessing or a curse. You essentially start on C level.
So fundamental things like IPC (inter process communication) is something you have to deal with yourself. If you want multi-tenancy that is likewise not supported out of the box. And all of this is before we venture into protocol standards, message formats and async vs synchronous execution.
The idea behind Remoting SDK is to get away from this style of low-level hacking. Without sounding negative, it provides the missing pieces that Delphi lacks, including the stuff that C# developers enjoy under .net (and then some). So if you are a Delphi developer who look over at C# with smudge of envy, then you are going to love Remoting SDK.
Say goodbye to boilerplate mistakes
Writing distributed servers and services is boring work. For each function you expose, you have to define the parameters and data-types in a portable way, then you have to implement the code that represents the exposed function and finally the interface itself that can be consumed by clients. The latter must be defined in a way that works with other languages too, not just Delphi. So while server tech in it’s essential form is quite simple, it’s the infrastructure that sets the stage of how quickly you can apply improvements and adapt to change.
For example, let’s say you have implemented a wonderful new service. It exposes 60 awesome functions that your customers can consume in their own work. The amount of boilerplate code for 60 distributed functions, especially if you operate with composite data types, is horrendous. It is a nightmare to manage and opens up for sloppy, unnecessary mistakes.
This is where Remoting SDK truly shines. When you install the software, it integrates it’s editors and wizards closely with the Delphi IDE. It adds a ton of new project types, components and whatnot – but the most important feature is without a doubt the service designer.

Start the service-designer in any server or service project and you can edit the methods, data types and interfaces your system expose to the world
As the name implies, the service designer allows you to visually define your services. Adding a new function is a simple click, the same goes for datatypes and structures (record types). These datatypes are exposed too and can be consumed from any modern language. So a service you make in Delphi can be used from C#, C/C++, Java, Oxygene, Swift (and visa-versa).
Auto generated code
A service designer is all good and well I hear you say, but what about that boilerplate code? Well Remoting SDK takes care of that too (kinda the point). Whenever you edit your services, the designer will auto-generate a new interface unit for you. This contains the classes and definitions that describe your service. It will also generate an implementation unit, with empty functions; you just need to fill in the blanks.
The designer is also smart enough not to remove code. So if you go in and change something, it won’t just delete the older implementation procedure. Only the params and names will be changed if you have already written some code.

Having changed a service, hitting F9 re-generates the interface code automatically. Your only job is to fill in the code for each method in the implementation units. The SDK takes care of everything else for you
The service information, including the type information, is stored in a special file format called “rodl”. This format is very close to Microsoft WSDL format, but it holds more information. It’s important to underline that you can import the service directly from your servers (optional naturally) as WSDL. So if you want to consume a Remoting SDK service using Delphi’s ordinary RIO components, that is not a problem. Visual Studio likewise imports and consumes services – so Remoting SDK behaves identical regardless of platform or language used.
Remoting SDK is not just for Delphi, just to be clear on that. If you are presently using both Delphi and C# (which is a common situation), you can buy a license for both C# and Delphi and use whatever language you feel is best for a particular task or service. You can even get Remoting SDK for Javascript and call your service-stack directly from your website if you like. So there are a lot of options for leveraging the technology.
Transport is not content
OK so Remoting SDK makes it easy to define distributed services and servers. But what about communication? Are we boxed into RemObjects way of doing things?
The remoting framework comes with a ton of components, divided into 3 primary groups:
- Servers
- Channels (clients)
- Messages
The reason for this distinction is simple: the ability to transport data, is never the same as the ability to describe data. For example, a message is always connected to a standard. It’s job is ultimately to serialize (represent) and de-serialize data according to a format. The server’s job is to receive a request and send a response. So these concepts are neatly decoupled for maximum agility.
As of writing the SDK offers the following message formats:
- Binary
- Post
- SOAP
- JSON
If you are exposing a service that will be consumed from JavaScript, throwing in a TROJSONMessage component is the way to go. If you expect messages to be posted from your website using ordinary web forms, then TROPostMessage is a perfect match. If you want XML then TROSOAPMessage rocks, and if you want fast, binary messages – well then there is TROBinaryMessage.
What you must understand is that you don’t have to pick just one! You can drop all 4 of these message formats and hook them up to your server or channel. The SDK is smart enough to recognize the format and use the correct component for serialization. So creating a distributed service that can be consumed from all major platforms is a matter of dropping components and setting a property.

If you double-click on a server or channel, you can link message components with a simple click. No messy code snippets in sight.
Multi-tenancy out of the box
With the release of Rad-Server as a part of Delphi, people have started to ask what exactly multi-tenancy is and why it matters. I have to be honest and say that yes, it does matter if you are creating a service stack where you want to isolate the logic for each customer in compartments – but the idea that this is somehow new or unique is not the case. Remoting SDK have given users multi-tenancy support for 15+ years, which is also why I haven’t been too enthusiastic with Rad-Server.
Now don’t get me wrong, I don’t have an axe to grind with Rad-Server. The only reason I mention it is because people have asked how i feel about it. The tech itself is absolutely welcome, but it’s the licensing and throwing Interbase in there that rubs me the wrong way. If it could run on SQLite3 and was free with Enterprise I would have felt different about it.

There are various models for multi-tenancy, but they revolve around the same principles
To get back on topic: multi-tenancy means that you can dynamically load services and expose them on demand. You can look at it as a form of plugin functionality. The idea in Rad-Server is that you can isolate a customer’s service in a separate package – and then load the package into your server whenever you need it.
The reason I dislike Rad-Server in this respect, is because they force you to compile with packages. So if you want to write a Rad-Server system, you have to compile your entire project as package-based, and ship a ton of .dpk files with your system. Packages is not wrong or bad per-se, but they open your system up on a fundamental level. There is nothing stopping a customer from rolling his own spoof package and potentially bypass your security.
There is also an issue with un-loading a package, where right now the package remains in memory. This means that hot-swapping packages without killing the server wont work.
Rad-Server is also hardcoded to use Interbase, which suddenly bring in licensing issues that rubs people the wrong way. Considering the price of Delphi in 2019, Rad-Server stands out as a bit of an oddity. And hardcoding a database into it, with the licensing issues that brings -just rendered the whole system mute for me. Why should I pay more to get less? Especially when I have been using multi-tenancy with RemObjects for some 15 years?
With Remoting SDK you have something called DLL servers, which does the exact same thing – but using ordinary DLL files (not packages!). You don’t have to compile your system with packages, and it takes just one line of code to make your main dispatcher aware of the loaded service.
This actually works so well that I use Remoting SDK as my primary “plugin” system. Even when I write ordinary desktop applications that has nothing to do with servers or services – I always try to compartmentalize features that could be replaced in the future.
For example, I’m a huge fan of ElevateDB, which is a native Delphi database engine that compiles directly into your executable. By isolating that inside a DLL as a service, my application is now engine agnostic – and I get a break from buying a truck load of components every time Delphi is updated.
Saving money
The thing about DLL services, is that you can save a lot of money. I’m actually using an ElevateDB license that was for Delphi 2007. I compiled the engine using D2007 into a DLL service — and then I consume that DLL from my more modern Delphi editions. I have no problem supporting or paying for components, that is right and fair, but having to buy new licenses for every single component each time Delphi is updated? This is unheard of in other languages, and I would rather ditch the platform all together than forking out $10k ever time I update.

A DLL server can be used for many things if you are creative about it
While we are on the subject – Hydra is another great money saver. It allows you to use .net and Java libraries (both visual and non-visual) with Delphi. With Hydra you can design something in .net, compile it into a DLL file, and then use that from Delphi.
But — you can also compile things from Delphi, and use it in newer versions of Delphi. Im not forking out for a Developer Express update just to use what I have already paid for in the latest Delphi. I have one license, I compile the forms and components into a Hydra Module — and then use it from newer Delphi editions.

Hydra, which is a separate product, allows you to stuff visual components and forms inside a vanilla DLL. It allows cross language use, so you can finally use Java and .net components inside your Delphi application
Bonjour support
Another feature I love is the zero configuration support. This is one of those things that you often forget, but that suddenly becomes important once you deploy a service stack on cluster level.
Remoting SDK comes with support for Apple Bonjour, so if you want to use that functionality you have to install the Bonjour library from Apple. Once installed on your host machines, your RemObjects services can find each other.
ZeroConfig is not that hard to code manually. You can roll your own using UDP or vanilla messages. But getting service discovery right can be fiddly. One thing is broadcasting an UDP message saying “here I am”, it’s something else entirely to allow service discovery on cluster level.
If Bonjour is not your cup of tea, the SDK provides a second option, which is RemObjects own zero-config hub. You can dig into the documentation to find out more about this.
What about that IPC stuff you mentioned?
I mentioned IPC (inter process communication) at the beginning here, which is a must have if you are making a service stack where each member is expected to talk to the others. In a large server-system the services might not exist on the same, physical hardware either, so you want to take height for that.
With the SDK this is just another service. It takes 10 minutes to create a DLL server with the functionality to send and receive messages – and then you just load and plug that into all your services. Done. Finished.
Interestingly, Remoting SDK supports named-pipes. So if you are running on a Windows network it’s even easier. Personally I prefer to use a vanilla TCP/IP based server and channel, that way I can make use of my Linux blades too.
Building on the system
There is nothing stopping you from expanding the system that RemObjects have established. You are not forced to only use their server types, message types and class framework. You can mix and match as you see fit – and also inherit out your own variation if you need something special.
For example, WebSocket is an emerging standard that has become wildly popular. Remoting SDK does not support that out of the box, the reason is that the standard is practically identical to the RemObjects super-server, and partly because there must be room for third party vendors.
Andre Mussche took the time to implement a WebSocket server for Remoting SDK a few years back. Demonstrating in the process just how easy it is to build on the existing infrastructure. If you are already using Remoting SDK or want WebSocket support, head over to his github repository and grab the code there: https://github.com/andremussche/DelphiWebsockets
I could probably write a whole book covering this framework. For the past 15 years, RemObjects Remoting SDK is the first product I install after Delphi. It has become standard for me and remains an integral part of my toolkit. Other packages have come and gone, but this one remains.
Hopefully this post has tickled your interest in the product. No matter if you are maintaining a legacy service stack, or thinking about re implementing your existing system in something future-proof, this framework will make your life much, much easier. And it wont break the bank either.
You can visit the product page here: https://www.remotingsdk.com/ro/default.aspx
And you can check out the documentation here: https://docs.remotingsdk.com/
RemObjects Remoting SDK?
Reading this you could be forgiven for thinking that I must promote RemObjects products, It’s my job now right? Well yes, but also no.
The thing is, I’m really not “traveling salesman” material by any stretch of the imagination. My tolerance for bullshit is ridiculously low, and being practical of nature I loath fancy products that cost a fortune yet deliver nothing but superficial fluff.
The reasons I went to work at RemObjects are many, but most of all it’s because I have been an avid supporter of their products since they launched. I have used and seen their products in action under intense pressure, and I have come to put some faith in their solutions.
Trying to describe what it’s like to write servers that should handle thousands of active user “with or without” RemObjects Remoting SDK is exhausting, because you end up sounding like a fanatic. Having said that, I feel comfortable talking about the products because I speak from experience.
I will try to outline some of the benefits here, but you really should check it out yourself. You can download a trial directly here: https://www.remotingsdk.com/ro/
Remoting framework, what’s that?
RemObjects Remoting framework (or “RemObjects SDK” as it was called earlier) is a framework for writing large-scale RPC (remote procedure call) servers and services. Unlike the typical solutions available for Delphi and C++ builder, including those from Embarcadero I might add, RemObjects framework stands out because it distinguishes between transport, host and message-format – and above all, it’s sheer quality and ease of use.
This separation between transport, host and message-format makes a lot of sense, because the parameters and data involved in calling a server-method, shouldn’t really be affected by how it got there.
And this is where the fun begins because the framework offers you a great deal of different server types (channels) and you can put together some interesting combinations by just dragging and dropping components.
How about JSON over email? Or XML over pipes?
The whole idea here is that you don’t have to just work with one standard (and pay through the nose for the privilege). You can mix and match from a rich palette of transport mediums and message-formats and instead focus on your job; to deliver a kick-ass product.
And should you need something special that isn’t covered by the existing components, inheriting out your own channel or message classes is likewise a breeze. For example, Andre Mussche have some additional components on GitHub that adds a WebSocket server and client. So there is a lot of room for expanding and building on the foundation provided by RemObjects.
And this is where RemObjects has the biggest edge (imho), namely that their solutions shaves weeks if not months off your development time. And the central aspect of that is their integrated service designer.
Integration into the Delphi IDE
Dropping components on a form is all good and well, but the moment you start coding services that deploy complex data-types (records or structures) the amount of boilerplate code can become overwhelming.
The whole point of a remoting framework is that it should expose your services to the world. Someone working in .net or Java on the other side of the planet should be able to connect, consume and invoke your services. And for that to happen every minute detail of your service has to follow standards.
When you install RemObjects SDK, it also integrates into the Delphi IDE. And one of the features it integrates is a complete, separate service designer. The designer can also be used outside of the Delphi IDE, but I cannot underline enough how handy it is to be able to design your services visually, right there and then, in the Delphi IDE.
This designer doesn’t just help you design your service description (RemObjects has their own RODL file-format, which is a bit like a Microsoft WSDL file), the core purpose is to auto-generate all the boilerplate code for you — directly into your Delphi project (!)
So instead of you having to spend a week typing boilerplate code for your killer solution, you get to focus on implementing the actual methods (which is what you are supposed to be doing in the first place).
DLL services, code re-use and multi-tenancy
The idea of multi-tenancy is an interesting one. One that I talked about with regards to Rad-Server both in Oslo and London before christmas. But Rad-Server is not the only system that allows for multi-tenancy. I was doing multi-tenancy with RemObjects SDK some 14 years ago (if not earlier).
Remember how I said the framework distinguishes between transport, message and host? That last bit, namely host, is going to change how you write applications.
When you install the framework, it registers a series of custom project types inside the Delphi IDE. So if you want to create a brand new RemObjects SDK server project, you can just do that via the ordinary File->New->Other menu option.
One of the project types is called a DLL Server. Which literally means you get to isolate a whole service library inside a single DLL file! You can then load in this DLL file and call the functions from other projects. And that is, ultimately, the fundamental principle for multi-tenancy.
And no, you don’t have to compile your project with external packages for this to work. The term “dll-server” can also be a bit confusing, because we are not compiling a network server into a DLL file, we are placing the code for a service into a DLL file. I used this project type to isolate common code, so I wouldn’t have to copy unit-files all over the place when delivering the same functionality.
It’s also a great way to save money. Don’t want to pay for that new upgrade? Happy with the database components you have? Isolate them in a DLL-Server and continue to use the code from your new Delphi edition. I have Delphi XE3 Database components running inside a RemObjects DLL-Server that I use from Delphi XE 10.3.
In my example I was doing business-logic for our biggest customers. Each of them used the same database, but they way they registered data was different. The company I worked for had bought up these projects (and thus their customers with them), and in order to keep the customers happy we couldn’t force them to re-code their systems to match ours. So we had to come up with a way to upgrade our technology without forcing a change on them.
The first thing I did was to create a “DLL server” that dealt with the database. It exposed methods like openTable(), createInvoice(), getInvoiceById() and so on. All the functions I would need to work with the data without getting my fingers dirty with SQL outside the DLL. So all the nitty gritty of SQL components, queries and whatnot was neatly isolated in that DLL file.
I then created separate DLL-Server projects for each customer, implemented their service interfaces identical to their older API. These DLL’s directly referenced the database library for authentication and doing the actual work.

When integrated with the IDE, you are greeted with a nice welcome window when you start Delphi. Here you can open examples or check out the documentation
Finally, I wrapped it all up in a traditional Windows system service, which contained two different server-channels and the message formats they needed. When the service was started it would simply load in the DLL’s and manually register their services and types with the central channel — and voila, it worked like a charm!
Rock solid
Some 10 years after I delivered the RemObjects based solution outlined above, I got a call from my old employer. They had been victim of a devastating cyber attack. I got a bit anxious as he went on and on about damages and costs, fearing that I had somehow contributed to the situation.
But it turned out he called to congratulate me! Out of all the services in their server-park, mine were the only ones left standing when the dust settled.
The RemObjects payload balancer had correctly dealt with both DDOS and brute force attacks, and the hackers were left wanting at the gates.
LDef parser done
Note: For a quick introduction to LDef click here: Introduction to LDef.
Great news guys! I finally finished the parser and model builder for LDef!
That means we just need to get the assembler ported. This is presently running fine under Smart Pascal (I like to prototype things there since its faster) – and it will be easy to port it over to Delphi and Freepascal after the model has gone through the steps.
I’m really excited about this project and while I sadly don’t have much free time – this is a project I truly enjoy working on. Perhaps not as much as Smart Pascal which is my baby, but still; its turning into a fantastic system.
Thoughts on the architecture
One of the things I added support for, and that I have hoped that Embarcadero would add to Delphi for a number of years now, is support for contract coding. This is a huge topic that I’m not jumping into here, but one of the features it requires is support for entry and exit sections. Essentially that you can define code that executes before the method body and directly after it has finished (before the result is returned if it’s a function).
This opens up for some very clever means of preventing errors, or at the very least give the user better information about what went wrong. Automated tests also benefits greatly from this.
For example, a normal object pascal method looks, for example, like this:
procedure TForm1.MySpecialMethod; begin writeln("You called my-special-method") end;
The basis of contract design builds on the classical and expands it as such:
procedure TForm1.MySpecialMethod; Before() begin writeln("Before my-special-method"); end; After() begin writeln("After my-special-method"); end; begin writeln("You called my-special-method") end;
Note: contract design is a huge system and this is just a fragment of the full infrastructure.
What is cool about the before/after snippets, is that they allow you to verify parameters before the body is even executed, and likewise you get to work on the result before the value is returned (if any).
You mights ask, why not just write the tests directly like people do all the time? Well, that is true. But there will also be methods that you have no control over, like a wrapper method that calls a system library for instance. Being able to attach before/after code for externally defined procedures helps take the edge off error testing.
Secondly, if you are writing a remoting framework where variant data and multi-threaded invocation is involved – being able to check things as they are dispatched means catching potential errors faster – leading to better performance.
As always, coding techniques is a source of argument – so im not going into this now. I have added support for it and if people don’t need it then fine, just leave it be.
Under LDef assembly it looks like this:
public void main() { enter { } leave { } }
Well I guess that’s all for now. Hopefully my next LDef post will be about the assembler being ready – leaving just the linker. I need to experiment a bit with the codegen and linker before the unit format is complete.
The bytecode-format needs to include enough information so that the linker can glue things together. So every class, member, field etc. must be emitted in a way that is easy and allows the linker to quickly look things up. It also needs to write the actual, resulting method offsets into the bytecode.
Have a happy weekend!
LDef Intermediate Language
The LDEF bytecode engine is some time away from completion, but the IL source format that the assembler reads and turns into bytecode is getting there. At the moment there are only a few tidbits left to explore, like interfaces and generics, but those will be added.
It’s a real brain teaser because – some of the stuff that makes up a compileris not really related to code. When you think about generics you often make the mistake of thinking this is a code feature, like inheritance or virtual methods; it’s something that the code-emitter has to deal with or runtime engine to take height for. But generics is actually implemented higher up. It exists between the parser and code-emitter.
Interfaces is another mirage or technological illusion. When you work with classes and interfaces you get a sense that it’s a solid thing, you know – you write a class and create objects and people imagine these objects as independent, material objects in a virtual space. I tend to think of instances as molecules.
But objects is ultimately an illusion. Im not going to cover the intricate details of object orientation here, but OOP is actually about separating the data (fields) from the code acting on that data (members). So the only objects that really exist in the memory of a computer when you create instances, are buffers representing the fields of your class – combined with references and a list mapping the entrypoints for the members that makes up that instance (VMT). Compilers actually mimic object orientation by adding a secret parameter to all your methods, namely the “self” parameter. This “self” is a pointer to a record that contains all the information pertaining to that instance.
Which is really cool because then you can create as many instances as you like – and they all use the same code. Which ofcourse object orientation is all about. But there is no such thing as an independent instance floating around computer memory. That is an optical illusion of sorts.
LDEF virtual machine
Virtual machines get’s to have all the fun. I mean, writing a compiler that emits real machine code is in itself not that hard, but generating a scheme that makes object orientation work and that keeps track of everything is. The x86 cpu architecture may be powerful but it’s an absolute bitch to work with. It has few registers (compared to ARM and PPC), several instruction sets and is known to be a somewhat unfriendly place. This is the reason that compiler makers tend to stick to a golden mean of instructions. Compilers like C++ builder and Delphi could actually generate faster and more efficient code if they knew exactly what cpu and architecture you used. But since that’s not how PC’s work and there are some 20 different cpu models on the market at any given time – with huge differences between AMD and Intel, it makes sense to use a safe collection of instructions.
LDEF is a bytecode runtime engine. And one of the perks of bytecode is that it’s very much abstracted from the whole issue of real-life assembly. Most programmers think that – if you make bytecodes then you dont have to think about low level stuff, which is cheating. That may be the case for other languages and engines, but not LDEF. In fact the whole point of writing a new engine is because I wanted a bytecode format that was closer to machine code. This is very important because at some point myself or someone else will write a JIT compiler for this, and if the runtime is to high-level or abstract, that is going to be very hard.

LDEF testbed
The LDEF instruction-set represents a golden mean of the average processor instruction set. I have taken instructions that most processors have, typical stuff like add, subtract, divide, modulus, multiply (and so on). Testing is where I have given LDEF some advantages, for instance when testing a list of parameters for conditions. Instead of keeping that as two instructions (test, branch [condition]) I have isolated that in a single instruction.
Let’s look at the instruction set so far:
- move
- blit – move memory block
- valloc – allocate variable
- vfree – free variable
- malloc – allocate memory
- mfree – release memory
- add
- sub
- mul
- muldiv
- div
- mod
- moddiv
- lsr – logical bit-shift right
- lsl – logical bit-shift left
- cmp – compare
- tst -test
- bne – branch not equal
- beq – branch equal
- ble – branch less
- bgt – branch greater
- jsr – jump sub routine
- jmp – jump absolute
- push – push to stack
- pop – pop from stack
- IFnb – test and branch if not equal
- IFtb – test and branch if equal
- throw – cause exception
- syscall – invoke engine spesific, delphi code
One of the benefits of a virtual machine, is that you can do some magic with variables. In a real machinecode compiler, variable allocation and doing read / write operations can be very complex. But in a virtual machine you thankfully get to do something about that.
So LDEF allows you to move data seamlessly between variables and registers, which really is a massive time saver. The code standard also supports two levels of resources, global and local. The latter meaning class in this case. So there is ample room for high-level languages to store their data and implement classical features (like “resourcestring” in Delphi).
You also have support for constants, these are stored separately in the bytecode and loaded into a lookup-table associated with a class. Constants are different from resources in that they are designed to be quickly referenced. Constants has more infrastructure in the form of lookup-tables – because they are meant to be used with the instructions. Resources are more like resource-strings in Delphi, they are stored in their own place in the bytecode and are loaded on demand. Constants are not. They are loaded into memory and are instantly available.
Having said that, the bytecode compiler also supports in-place data. Meaning that you can chose to write constant data where they are used. So instead of having to load a string constant (for example) before working with it, you can compile the string directly into the move instruction (or any other instruction).So this is actually perfectly valid:
move r0, "Hello world";
Other virtual machines, like the Java engine, force you to do this in two steps, which is slower:
lda r0, cost_id; // reference const by id ldc r1, (r0); // get actual value into r1
You can also assign variables directly, you dont need to load the effective address first. So there is no extract cost involved in moving data between a register and variable, variable and variable, or resource / cost to a variable. This makes life much easier:
move V[$00F6], "String assignment"; move V[$01F0], 19875.32; move V[$196], V[$197];
Registers
Another thing that is beneficial is that LDEF has 32 registers to work with (you can actually change that, so if you need 64 that’s no problem). How you use these is up to you, but it gives a lot of room for OOP schemes. For instance, if a language has a maximum limit of 6 parameters per method – you actually dont need to pass values on the stack at all (like Java does) but you can map parameters directly to registers.
Delphi, C++ builder and other native solutions tend to use stack-schemes to store information. So a pointer to some list is stored as the fourth item on the stack, and some other entry on the third (and so on). Which is perfectly fine and very effective on a real CPU (you really have no choice on a real x86). In LDEF you can now use spesific registers instead, which is a lot easier. What scheme you chose to use if naturally up to you – but at least the option is there.
Here are the registers LDEF presently supports
- R0 – R31: general purpose registers
- SP: stack pointer
- PC: program control register
- BC: branch control register
- ES: exception state register
Optimization
If you are wondering why a virtual machine would support so much similar stuff, constants and resources, in place data or not, this all have to do with optimalization.
For example, if your compiler decides to collect all assignment values as constants, the codesize of your program might be smaller; it all depends on what your code looks like. You will naturally consolidate identical strings, integer values and so on. So even if you have a program that writes “Hello world!” 10.000 times to the console, only one “Hello world!” will be stored in the bytecode file. So constants gives you smaller code, but at a cost of speed – since constants needs a lookup whenever they are used.
Now constants here are not “cost” declarations. Constants on this level is actually the values you write in plain-text in your program, stuff like this:
FMyText := 'Welcome to my program'; FCount := 100;
Both “welcome to my program” and “100” are constants. These are values that will never change, and compilers usually have to deal with these values in one of two ways. Both of which I have explained above (in place or not).
Source format
The LDEF intermediate format looks pretty much like C++. C is fast and easier to parse than other languages and it made sense to pick that.But this similarity is paper thin, in that only the constructs of C++ is used. Methods only constains the LDEF assembly language code, and variables are isolated in Alloc sections. The virtual machine takes care of the OOP layer for you so you dont have to worry about that.
Here is an example to give you a feel for it:
#include <stdio>; #include <oop>; struct CustomType { uint32 ctMagic; uint32 ctSize; uint8 ctData[4096]; } resources { dc.s #welcome, "welcome to LDef"; } class TBaseObject: object { /* class-fields */ alloc { uint8 temp; uint32 counter; CustomType handle; } /* Parser now handles register mapping */ public void main(r0 as count, r1 as text) { enter { } leave { } alloc { /* method variables */ uint32 _longs; uint32 _singles; } /* calc longs */ move r1, count; mod r1, 8; move r2, count; move _longs, r1; /* calc singles */ move r3, r1; mul r3, 8; sub r2, r3; move _singles, r2 } /* test multi push to stack */ private void _cleanup() { push [r0, r1, r2]; } }
Keep in mind that the format is not 100% finished yet, there are still a few tidbits that needs to be worked out. But the general system is firmly in place.
One of the cool things is that I get to add a few missing features from my favorite language, Delphi (object pascal). If you look at the first method (main), you will notice that it has two extra sections: enter and leave.
These sections can contain code that execute directly before and after the body of the method. In pascal it would look something like this:
procedure TMyClass.SomeProc; before writeln('about to execute someproc'); end; after writeln('Someproc finished'); end; begin end;
The purpose of these, especially the before() section, is to make sure the parameters are valid. So before is used to check that the parameters are within a legal range. After is naturally used to ensure that the result (for functions) is valid.
It also opens up for some interesting logging and debugging aspects.
More to come
This is turning into a long rant, but hopefully you found it interesting. I’ll keep you posted as the engine progress. There is still some way to go, but we are getting there. Once LDEF is implemented the fun starts, thats when I’ll code high-level parsers that targets the engine. First stop is naturally and object pascal compiler 🙂
Build NodeJS from Source

nodeJS rocks!
Since your Smart Mobile Studio programs can run from Linux under Raspberry PI, I figured I could write a few words about building from C/C++ source. Normally you don’t need to do this, but in case the process-manager (which makes sure your Smart NodeJS servers can be clustered and essentially controlled as services) can be picky about the version – you may suddenly find yourself needing the cutting edge.
Open up a command-line prompt and CD (change dir) to the location you want to keep node. Since this is so essential to my system I keep it at root (/), but you may want to stuff it in your home folder.
Either way, cd to the location then do:
sudo apt-get install build-essential sudo apt-get install curl openssl libssl-dev
Odds are this is up to date (it was on my fresh Raspbian image). If not, let it finish (you may need to press “y” at some point) and let it work itself out.
You also need to install git (may also already be there):
sudo apt-get install git
With those in place, let’s do a clone of the node.js repository:
git clone https://github.com/nodejs/node.git
When its finished you should have a “nodejs” folder available. So we cd into that and do a configure:
cd node ./configure make sudo make install
Now building anything on the Raspberry PI is slow. So the “make” call will probably take 1.5 hours depending on your PI. If its overclocked you will probably get away with 45 minutes, but a bog standard Raspberry PI 3 is much slower.
When it’s all done, test it with
node --version
As of writing that prints out “8.0.0-pre” since im using the cutting edge pre-release version right now.
Now you can install npm (node package manager) and them pm2 and enjoy the show 🙂
3D mathematics [book]
It’s not often I promote books, but this time I’ll make an exception: Mathematics for 3d programming and computer graphics.

A book worth every penny, even if you dont use 3d graphics very often
No matter if you are a Delphi programmer, Smart Pascal, Freepascal, C# or C++; sooner or later you are going to have to dip your fingers into what we may call “primal coding”. That means coding that was established some time ago, and that have since been isolated and standardized in APIS. This means that if you want to learn it, you are faced with the fact that everyone is teaching you how to use the API — not how to make it or how it works behind the scenes!
3D graphics
Once in a while I go on a retro-computer rant (I know, I know) talking about the good ol’ days. But there is a reason for this! And a good one at that. I grew up when things like 3d graphics didn’t exist. There were no 3d graphics on the Commodore 64 or the Amiga 500. The 80’s and early 90’s were purely 2d. So I have been lucky and followed the evolution of these things long before they became “standard” and isolated away in API’s.
Somewhere around the mid 80’s there was a shift from “top-down 2d graphics” in games and demo coding. From vanilla squares to isometric tiles (actually the first game that used it was Qbert, released in 1982). So rather than building up a level with 32 x 32 pixel squares – you built up games with 128 degrees tilted blocks (or indeed, hexagon shaped tiles).
This was the beginning of “3D for the masses” as we know it because it added a sense of depth to the game world.

Qbert, 1982, isometric view
With isometric graphics you suddenly had to take this depth factor into account. This became quite important when testing collisions between sprites. And it didn’t take long before the classical “X,Y,Z” formulas to become established.
As always, these things already existed (3D computation was common even back in the 70s). But their use in everyday lives were extremely rare. Suddenly 3d went from being the domain of architects and scientists – to being written and deployed by kids in bedrooms and code-shops. This is where the european demo scene really came to our aid.
Back to school
This book is about the math. And it’s explained in such a way that you don’t have to be good in it. Rather than teaching you how to use OpenGL or Direct3D, this book teaches you the basics of 3D rotation, vectors, matrixes and how it all fits together.
Why is this useful? Because if you know something from scratch it makes you a better programmer. It’s like cars. Knowing how to drive is the most important thing, but a mechanic will always have a deeper insight into what the vehicle can and cannot do.

Every facet is explained both as theorem and practical example
This is the book you would want if you were to create OpenGL. Or like me, when you don’t really like math but want to brush up on old tricks. We used this in demo coding almost daily when I was 14-15 years old. But I have forgotten so much of it, and the information is quite hard to find in a clear, organized way.
Now I don’t expect anyone to want to write a new 3D engine, but 3D graphics is not just about that. Take something simple, like how an iPhone application transition between forms. Remember the cube effect? Looking at that effect and knowing some basic 3D formulas and voila, it’s not a big challenge to recreate it in Delphi, C++, C# or whatever language you enjoy the most.
I mean, adding OpenGL or WebGL dependencies just to spin a few cubes or position stuff in 3D space? That’s overkill. It’s actually less than 200 lines of code.
Well, im not going to rant on about this — this book is a keeper!
A bit expensive, but its one of those books that will never go out of date and the information inside is universal and timeless.
Enjoy!
You must be logged in to post a comment.