C/C++ porting, QTX and general status
C is a language that I used to play around with a lot back in the Amiga days. I think the last time I used a C compiler to write a library must have been in 1992 or something like that? I held on to my Amiga 1200 for as long as i could – but having fallen completely in love with Pascal, I eventually switched to x86 and went down the Turbo Pascal road.
Lately however, C++ developers have been asking for their own Developer group on Facebook. I run several groups on Facebook in the so-called “developer” family. So you have Delphi Developer, FPC Developer, Node.JS Developer and now – C++Builder developer. The groups more or less tend to themselves, and the node.js and FPC groups are presently being seeded (meaning, that the member count is being grown for a period).
The C++Builder group however, is having the same activity level as the Delphi group almost, thanks to some really good developers that post links, tips and help solve questions. I was also fortunate enough to have David Millington come on the Admin team. David is leading the C++Builder project, so his insight and knowledge of both language and product is exemplary. Just like Jim McKeeth, he is a wonderful resource for the community and chime in with answers to tricky questions whenever he has time to spare.
Getting back in the saddle
Having working some 30 years with Pascal and Object Pascal, 25 of those years in Delphi, C/C++ is never far away. I have an article on the subject that i’ve written for the Idera Community website, so I wont dig too deep into that here — but needless to say, Rad Studio consists of two languages: Object Pascal and C/C++, so no matter how much you love either language, the other is never far away.
So I figured it was time for this old dog to learn some new tricks! I have always said that it’s wise to learn a language immediately below and above your comfort zone. So if Delphi is your favorite language, then C/C++ is below you (meaning: more low level and complex). Above you are languages like JavaScript and C#. Learning JavaScript makes strategic sense (or use DWScript to compile Pascal to JavaScript like I do).
When I started out, the immediate language below Object Pascal was never C, but assembler. So for the longest time I turned to assembler whenever I needed a speed boost; graphics manipulation and processing pixels is especially a field where assembly makes all the difference.
But since C++Builder is indeed an integral part of Rad Studio, and Object Pascal and C/C++ so intimately connected (they have evolved side by side), why not enjoy both assembly and C right?
So I decided to jump back into the saddle and see what I could make of it.
C/C++ is not as hard as you think
While I’m not going to rehash the article I have already prepared for the Idera Community pages here, I do want to encourage people to give it a proper try. I have always said that if you know an archetypal language, you can easily pick up other languages, because the archetypal languages will benefit you for a lifetime. This has to do with archetypal languages operating according to how computers really work; as opposed to optimistic languages (a term from the DB work, optimistic locking), also called contextual languages, like C#, Java, JavaScript etc. are based on how human beings would like things to be.
So I now had a chance to put my money where my mouth is.
When I left C back in the early 90s, I never bothered with OOP. I mean, I used C purely for shared libraries anyways, while the actual programs were done in Pascal or a hybrid language called Blitz Basic. The latter compiled to razor sharp machine code, and you could use inline assembly – which I used a lot back then (very few programmers on those machines went without assembler, it was almost given that you could use 68k in some capacity).
Without ruining the article about to be published, I had a great time with C++Builder. It took a few hours to get my bearings, but since both the VCL and FMX frameworks are there – you can approach C/C++ just like you would Object Pascal. So it’s a matter of getting an overview really.
Needless to say, I’ll be porting a fair share of my libraries to C/C++ when I have time (those that makes sense under that paradigme). It’s always good to push yourself and there are plenty of subtle differences that I found useful.
Quartex Media Desktop
When I last wrote about QTX we were nearing the completion of the FileSystem and Task Management service. The prototype had all its file-handling directly in the core service (or server) which worked just fine — but it was linked to the Smart Pascal RTL. It has taken time to write a new RTL + a full multi-user, platform independent service stack and desktop (phew!) but we are seeing progress!
The filesystem service is now largely done! There are a few synchronous calls I want to get rid of, but thankfully my framework has both async and sync variations of all file procedures – so that is now finished.
To make that clearer: first I have to wrap and implement the functionality for the RTL. Once they are in the RTL, I can use those functions to build the service functions. So yeah, it’s been extremely elaborate — but thankfully it’s also become a rich, well organized codebase (both the RTL and the Quartex Media Desktop codebases) – so I think we are ready to get cracking on the core!
The core is still operating with the older API. So our next step is to remove that from the core and instead delegate calls to the filesystem to our new service. So the core will simply be reduced to a post-office or traffic officer if you like. Messages come in from the desktops, and the core delegates the messages to whatever service is in charge of them.
But, this also means that both the core and the desktop must use the new and fancy messages. And this is where I did something very clever.
While I was writing the service, I also write a client class to test (obviously). And the way the core works — means that the same client that the core use to talk to the services — can be used by the desktop as well.
So our work in the desktop to get file-access and drives running again, is to wrap the client in our TQTXDevice ancestor class. The desktop NEVER accesses the API directly. All it knows about are these device drivers (or object instances). Which is how we solve things like DropBox and Google Drive support. The desktop wont have the faintest clue that its using Dropbox, or copying files between a local disk and Google Drive for example — because it only communicates with these device classes.
Recursive stuff
One thing that sucked about node.js function for deleting a folder, is that it’s recursive parameter doesn’t work on Windows or OS X. So I had to implement a full recursive deletefolder routine manually. Not a big thing, but slightly more painful than expected under asynchronous execution. Thankfully, Object Pascal allows for inline defined procedures, so I didn’t have to isolate it in a separate class.
Here is some of the code, a tiny spec compared to the full shabam, but it gives you an idea of what life is like under async conditions:
unit service.file.core; interface {.$DEFINE DEBUG} const CNT_PREFS_DEFAULTPORT = 1883; CNT_PREFS_FILENAME = 'QTXTaskManager.preferences.ini'; CNT_PREFS_DBNAME = 'taskdata.db'; CNT_ZCONFIG_SERVICE_NAME = 'TaskManager'; uses qtx.sysutils, qtx.json, qtx.db, qtx.logfile, qtx.orm, qtx.time, qtx.node.os, qtx.node.sqlite3, qtx.node.zconfig, qtx.node.cluster, qtx.node.core, qtx.node.filesystem, qtx.node.filewalker, qtx.fileapi.core, qtx.network.service, qtx.network.udp, qtx.inifile, qtx.node.inifile, NodeJS.child_process, ragnarok.types, ragnarok.Server, ragnarok.messages.base, ragnarok.messages.factory, ragnarok.messages.network, service.base, service.dispatcher, service.file.messages; type TQTXTaskServiceFactory = class(TMessageFactory) protected procedure RegisterIntrinsic; override; end; TQTXFileWriteCB = procedure (TagValue: variant; Error: Exception); TQTXFileStateCB = procedure (TagValue: variant; Error: Exception); TQTXUnRegisterLocalDeviceCB = procedure (TagValue: variant; DiskName: string; Error: Exception); TQTXRegisterLocalDeviceCB = procedure (TagValue: variant; LocalPath: string; Error: Exception); TQTXFindDeviceCB = procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception); TQTXGetDisksCB = procedure (TagValue: variant; Devices: JDeviceList; Error: Exception); TQTXGetFileInfoCB = procedure (TagValue: variant; LocalName: string; Info: JStats; Error: Exception); TQTXGetTranslatePathCB = procedure (TagValue: variant; Original, Translated: string; Error: Exception); TQTXCheckDevicePathCB = procedure (TagValue: variant; PathName: string; Error: Exception); TQTXServerExecuteCB = procedure (TagValue: variant; Data: string; Error: Exception); TQTXTaskService = class(TRagnarokService) private FPrefs: TQTXIniFile; FLog: TQTXLogEmitter; FDatabase: TSQLite3Database; FZConfig: TQTXZConfigClient; FRegHandle: TQTXDispatchHandle; FRegCount: integer; procedure HandleGetDevices(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleGetDeviceByName(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleCreateLocalDevice(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleDestroyDevice(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleFileRead(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleFileReadPartial(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleGetFileInfo(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleFileDelete(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleFileWrite(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleFileWritePartial(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleFileRename(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleGetDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleMkDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure HandleRmDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); procedure ExecuteExternalJS(Params: array of string; TagValue: variant; const CB: TQTXServerExecuteCB); procedure SendError(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage; Message: string); protected function GetFactory: TMessageFactory; override; procedure SetupPreferences(const CB: TRagnarokServiceCB); procedure SetupLogfile(LogFileName: string;const CB: TRagnarokServiceCB); procedure SetupDatabase(const CB: TRagnarokServiceCB); procedure ValidateLocalDiskName(TagValue: variant; Username, DeviceName: string; CB: TQTXCheckDevicePathCB); procedure RegisterLocalDevice(TagValue: variant; Username, DiskName: string; CB: TQTXRegisterLocalDeviceCB); procedure UnRegisterLocalDevice(TagValue: variant; UserName, DiskName:string; CB: TQTXUnRegisterLocalDeviceCB); procedure GetDevicesForUser(TagValue: variant; UserName: string; CB: TQTXGetDisksCB); procedure FindDeviceByName(TagValue: variant; UserName, DiskName: string; CB: TQTXFindDeviceCB); procedure FindDeviceByType(TagValue: variant; UserName: string; &Type: JDeviceType; CB: TQTXGetDisksCB); procedure GetTranslatedPathFor(TagValue: variant; Username, FullPath: string; CB: TQTXGetTranslatePathCB); procedure GetFileInfo(TagValue: variant; UserName: string; FullPath: string; CB: TQTXGetFileInfoCB); procedure SetupTaskTable(const TagValue: variant; const CB: TRagnarokServiceCB); procedure SetupOperationsTable(const TagValue: variant; const CB: TRagnarokServiceCB); procedure SetupDeviceTable(const TagValue: variant; const CB: TRagnarokServiceCB); procedure AfterServerStarted; override; procedure BeforeServerStopped; override; procedure Dispatch(Socket: TNJWebSocketSocket; Message: TQTXBaseMessage); override; public property Preferences: TQTXIniFile read FPrefs; property Database: TSQLite3Database read FDatabase; procedure SetupService(const CB: TRagnarokServiceCB); constructor Create; override; destructor Destroy; override; end; implementation //############################################################################# // TQTXFileenticationFactory //############################################################################# procedure TQTXTaskServiceFactory.RegisterIntrinsic; begin writeln("Registering task interface"); &Register(TQTXFileGetDeviceListRequest); &Register(TQTXFileGetDeviceByNameRequest); &Register(TQTXFileCreateLocalDeviceRequest); &Register(TQTXFileDestroyDeviceRequest); &Register(TQTXFileReadPartialRequest); &Register(TQTXFileReadRequest); &Register(TQTXFileWritePartialRequest); &Register(TQTXFileWriteRequest); &Register(TQTXFileDeleteRequest); &Register(TQTXFileRenameRequest); &Register(TQTXFileInfoRequest); &Register(TQTXFileDirRequest); &Register(TQTXMkDirRequest); &Register(TQTXRmDirRequest); &Register(TQTXFileRenameRequest); &Register(TQTXFileDirRequest); end; //############################################################################# // TQTXTaskService //############################################################################# constructor TQTXTaskService.Create; begin inherited Create; FPrefs := TQTXIniFile.Create(); FLog := TQTXLogEmitter.Create(); FDatabase := TSQLite3Database.Create(nil); FZConfig := TQTXZConfigClient.Create(); FZConfig.Port := 2292; self.OnUserSignedOff := procedure (Sender: TObject; Username: string) begin WriteToLogF("We got a service signal! User [%s] has signed off completely", [Username]); end; MessageDispatch.RegisterMessage(TQTXFileGetDeviceListRequest, @HandleGetDevices); MessageDispatch.RegisterMessage(TQTXFileGetDeviceByNameRequest, @HandleGetDeviceByName); MessageDispatch.RegisterMessage(TQTXFileCreateLocalDeviceRequest, @HandleCreateLocalDevice); MessageDispatch.RegisterMessage(TQTXFileDestroyDeviceRequest, @HandleDestroyDevice); MessageDispatch.RegisterMessage(TQTXFileReadRequest, @HandleFileRead); MessageDispatch.RegisterMessage(TQTXFileReadPartialRequest, @HandleFileReadPartial); MessageDispatch.RegisterMessage(TQTXFileWriteRequest, @HandleFileWrite); MessageDispatch.RegisterMessage(TQTXFileWritePartialRequest, @HandleFileWritePartial); MessageDispatch.RegisterMessage(TQTXFileInfoRequest, @HandleGetFileInfo); MessageDispatch.RegisterMessage(TQTXFileDeleteRequest, @HandleFileDelete); MessageDispatch.RegisterMessage(TQTXMkDirRequest, @HandleMkDir); MessageDispatch.RegisterMessage(TQTXRmDirRequest, @HandleRmDir); MessageDispatch.RegisterMessage(TQTXFileRenameRequest, @HandleFileRename); MessageDispatch.RegisterMessage(TQTXFileDirRequest, @HandleGetDir); end; destructor TQTXTaskService.Destroy; begin // decouple logger from our instance self.logging := nil; // Release prefs + log FPrefs.free; FLog.free; FZConfig.free; inherited; end; procedure TQTXTaskService.SendError(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage; Message: string); begin var reply := TQTXErrorMessage.Create(request.ticket); try reply.Code := CNT_MESSAGE_CODE_ERROR; reply.Routing.TagValue := Request.Routing.TagValue; reply.Response := Message; if Socket.ReadyState = rsOpen then begin try Socket.Send( reply.Serialize() ); except on e: exception do WriteToLog(e.message); end; end else WriteToLog("Failed to dispatch error, socket is closed error"); finally reply.free; end; end; procedure TQTXTaskService.ExecuteExternalJS(Params: array of string; TagValue: variant; const CB: TQTXServerExecuteCB); begin var LTask: JChildProcess; var lOpts := TVariant.CreateObject(); lOpts.shell := false; lOpts.detached := true; Params.insert(0, '--no-warnings'); // Spawn a new process, this creates a new shell interface try LTask := child_process().spawn('node', Params, lOpts ); except on e: exception do begin if assigned(CB) then CB(TagValue, e.message, e); exit; end; end; // Map general errors on process level LTask.on('error', procedure (error: variant) begin {$IFDEF DEBUG} writeln("error->" + error.toString()); {$ENDIF} WriteToLog(error.toString()); if assigned(CB) then CB(TagValue, "", nil); end); // map stdout so we capture the output LTask.stdout.on('data', procedure (data: variant) begin if assigned(CB) then CB(TagValue, data.toString(), nil); end); // map stderr so we can capture exception messages LTask.stderr.on('data', procedure (error:variant) begin {$IFDEF DEBUG} writeln("stdErr->" + error.toString()); {$ENDIF} if assigned(CB) then CB(TagValue, "", nil); WriteToLog(error.toString()); end); end; function TQTXTaskService.GetFactory: TMessageFactory; begin result := TQTXTaskServiceFactory.Create(); end; procedure TQTXTaskService.SetupService(const CB: TRagnarokServiceCB); begin SetupPreferences( procedure (Error: Exception) begin // No logfile yet setup (!) if Error nil then begin WriteToLog("Preferences setup: Failed!"); WriteToLog(error.message); raise error; end else WriteToLog("Preferences setup: OK"); // logfile-name is always relative to the executable var LLogFileName := TQTXNodeFileUtils.IncludeTrailingPathDelimiter( TQTXNodeFileUtils.GetCurrentDirectory ); LLogFileName += FPrefs.ReadString('log', 'logfile', 'log.txt'); // Port is defined in the ancestor, so we assigns it here Port := FPrefs.ReadInteger('networking', 'port', CNT_PREFS_DEFAULTPORT); SetupLogfile(LLogFileName, procedure (Error: Exception) begin if Error nil then begin WriteToLog("Logfile setup: Failed!"); WriteToLog(error.message); raise error; end else WriteToLog("Logfile setup: OK"); SetupDatabase( procedure (Error: Exception) begin if Error nil then begin WriteToLog("Database setup: Failed!"); WriteToLog(error.message); if assigned(CB) then CB(Error) else raise Error; end else WriteToLog("Database setup: OK"); if assigned(CB) then CB(nil); end); end); end); end; procedure TQTXTaskService.SetupPreferences(const CB: TRagnarokServiceCB); begin var lBasePath := TQTXNodeFileUtils.GetCurrentDirectory; var LPrefsFile := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(LBasePath) + CNT_PREFS_FILENAME; if TQTXNodeFileUtils.FileExists(LPrefsFile) then begin FPrefs.LoadFromFile(nil, LPrefsFile, procedure (TagValue: variant; Error: Exception) begin if Error nil then begin if assigned(CB) then CB(Error) else raise Error; exit; end; if assigned(CB) then CB(nil); end); end else begin var LError := Exception.Create('Could not locate preferences file: ' + LPrefsFile); WriteToLog(LError.message); if assigned(CB) then CB(LError) else raise LError; end; end; procedure TQTXTaskService.SetupLogfile(LogFileName: string;const CB: TRagnarokServiceCB); begin // Attempt to open logfile // Note: Log-object error options is set to throw exceptions try FLog.Open(LogFileName); except on e: exception do begin if assigned(CB) then begin CB(e); exit; end else begin writeln(e.message); raise; end; end; end; // We inherit from TQTXLogObject, which means we can pipe // all errors etc directly using built-in functions. So here // we simply glue our instance to the log-file, and its all good self.Logging := FLog as IQTXLogClient; if assigned(CB) then CB(nil); end; procedure TQTXTaskService.FindDeviceByType(TagValue: variant; UserName: string; &Type: JDeviceType; CB: TQTXGetDisksCB); begin UserName := username.trim().ToLower(); if Username.length < 1 then begin WriteToLog("Failed to lookup disk, username was invalid error"); var lError := EException.Create("Failed to lookup devices, invalid username"); if assigned(CB) then CB(TagValue, nil, lError) else raise lError; exit; end; GetDevicesForUser(TagValue, Username, procedure (TagValue: variant; Devices: JDeviceList; Error: Exception) begin if Error nil then begin if assigned(CB) then CB(TagValue, nil, Error) else raise Error; exit; end; var x := 0; while x < Devices.dlDrives.Count do begin if Devices.dlDrives[x].&Type &Type then begin Devices.dlDrives.delete(x, 1); continue; end; inc(x); end; if assigned(CB) then CB(TagValue, Devices, nil); end); end; procedure TQTXTaskService.FindDeviceByName(TagValue: variant; Username, DiskName: string; CB: TQTXFindDeviceCB); begin UserName := username.trim().ToLower(); if Username.length < 1 then begin var lLogText := "Failed to lookup device, username was invalid error"; WriteToLog(lLogText); var lError := EException.Create(lLogText); if assigned(CB) then CB(TagValue, nil, lError) else raise lError; exit; end; DiskName := DiskName.trim(); if DiskName.length < 1 then begin var lLogText := "Failed to lookup device, disk-name was invalid error"; WriteToLog(lLogText); var lError := EException.Create(lLogText); if assigned(CB) then CB(TagValue, nil, lError) else raise lError; exit; end; GetDevicesForUser(TagValue, Username, procedure (TagValue: variant; Devices: JDeviceList; Error: Exception) begin if Error nil then begin if assigned(CB) then CB(TagValue, nil, Error) else raise Error; exit; end; DiskName := DiskName.trim().ToLower(); var lDiskInfo: JDeviceInfo := nil; for var disk in Devices.dlDrives do begin if disk.Name.ToLower() = DiskName then begin lDiskInfo := disk; break; end; end; if assigned(CB) then CB(TagValue, lDiskInfo, nil); end); end; procedure TQTXTaskService.GetTranslatedPathFor(TagValue: variant; Username, FullPath: string; CB: TQTXGetTranslatePathCB); begin var lParser := TQTXPathParser.Create(); try var lInfo: TQTXPathData; if lparser.Parse(FullPath, lInfo) then begin // Locate the device for the path belonging to the user FindDeviceByName(TagValue, UserName, lInfo.MountPart, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin if Error nil then begin if assigned(CB) then CB(TagValue, FullPath, '', Error) else raise Error; exit; end; if Device.&Type dtLocal then begin var lError := EException.CreateFmt('Failed to translate path, device [%s] is not local error', [Device.Name]); if assigned(CB) then CB(TagValue, FullPath, '', Error) else raise Error; exit; end; // We want the path + filename, so we can append that to // the actual localized filesystem var lExtract := FullPath; delete(lExtract, 1, lInfo.MountPart.Length + 1); // Construct complete storage location var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices'; lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + lExtract; // Return translated path if assigned(CB) then CB(TagValue, FullPath, lFullPath, nil); end); end else begin var lErr := EException.CreateFmt("Invalid path [%s] error", [FullPath]); if assigned(CB) then CB(TagValue, FullPath, '', lErr) else raise lErr; end; finally lParser.free; end; end; procedure TQTXTaskService.GetFileInfo(TagValue: variant; UserName, FullPath: string; CB: TQTXGetFileInfoCB); begin var lParser := TQTXPathParser.Create(); try var lInfo: TQTXPathData; if lparser.Parse(FullPath, lInfo) then begin // Locate the device for the path belonging to the user FindDeviceByName(TagValue, UserName, lInfo.MountPart, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin if Error nil then begin if assigned(CB) then CB(TagValue, '', nil, Error) else raise Error; exit; end; case Device.&Type of dtLocal: begin // We want the path + filename, so we can append that to // the actual localized filesystem var lExtract := FullPath; delete(lExtract, 1, lInfo.MountPart.Length + 1); // Construct complete storage location var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices'; lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + lExtract; // Call the underlying OS to get the file statistics NodeJsFsAPI().lStat(lFullPath, procedure (Error: JError; Stats: JStats) begin if Error nil then begin var lError := EException.Create(Error.message); if assigned(CB) then CB(TagValue, lFullPath, nil, lError) else raise lError; exit; end; // And deliver if assigned(CB) then CB(TagValue, lFullPath, Stats, nil); end); end; dtDropbox, dtGoogle, dtMsDrive: begin var lError := EException.Create("Cloud bindings not activated error"); if assigned(CB) then CB(TagValue, '', nil, lError) end; end; end); end else begin var lErr := EException.CreateFmt("Invalid path [%s] error", [FullPath]); if assigned(CB) then CB(TagValue, '', nil, lErr) else raise lErr; end; finally lParser.free; end; end; procedure TQTXTaskService.GetDevicesForUser(TagValue: variant; Username: string; CB: TQTXGetDisksCB); begin UserName := username.trim().ToLower(); if Username.length < 1 then begin WriteToLog("Failed to lookup devices, username was invalid error"); var lError := EException.Create("Failed to lookup devices, invalid username"); if assigned(CB) then CB(TagValue, nil, lError) else raise lError; exit; end; var lTransaction: TQTXReadTransaction; if not TSQLite3Database(DataBase).CreateReadTransaction(lTransaction) then begin var lErr := EException.Create("Failed to create read-transaction error"); if assigned(cb) then CB(TagValue, nil, lErr) else raise lErr; exit; end; var lQuery := TSQLite3ReadTransaction(lTransaction); lQuery.SQL := "select * from devices where owner=?"; lQuery.Parameters.AddValueOnly(Username); lQuery.Execute( procedure (Sender: TObject; Error: Exception) begin if Error nil then begin if assigned(CB) then CB(TagValue, nil, Error) else raise Error; exit; end; var lDisks := new JDeviceList(); lDisks.dlUser := UserName; for var x := 0 to lQuery.datarows.length-1 do begin var lInfo := new JDeviceInfo(); lInfo.Name := lQuery.datarows[x]["name"]; lInfo.&Type := JDeviceType( lQuery.datarows[x]["type"] ); lInfo.owner := lQuery.datarows[x]["owner"]; lInfo.location := lQuery.datarows[x]["location"]; lInfo.APIKey := lQuery.datarows[x]["apikey"]; lInfo.APISecret := lQuery.datarows[x]["apisecret"]; lInfo.APIPassword := lQuery.datarows[x]["apipassword"]; lInfo.APIUser := lQuery.datarows[x]["apiuser"]; lDisks.dlDrives.add(lInfo); end; try if assigned(CB) then CB(TagValue, lDisks, nil); finally lQuery.free; end; end); end; procedure TQTXTaskService.ValidateLocalDiskName(TagValue: variant; Username, DeviceName: string; CB: TQTXCheckDevicePathCB); begin var Filename := 'disk.' + username + '.' + DeviceName + '.' + ord(JDeviceType.dtLocal).ToString(); var LBasePath := TQTXNodeFileUtils.GetCurrentDirectory(); LBasePath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(LBasePath) + 'userdevices'; // Make sure the device folder is there if not TQTXNodeFileUtils.DirectoryExists(LBasePath) then begin var lError := EException.CreateFmt("Directory not found: %s", [lBasePath]); if assigned(CB) then CB(TagValue, '', lError) else raise lError; exit; end; lBasePath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(LBasePath) + Filename; if TQTXNodeFileUtils.DirectoryExists(LBasePath) then begin var lError := EException.CreateFmt("Path already exist error [%s]", [lBasePath]); if assigned(CB) then CB(TagValue, '', lError) else raise lError; exit; end; // OK, folder is not created yet, so its good to go if assigned(CB) then CB(TagValue, Filename, nil); end; procedure TQTXTaskService.UnRegisterLocalDevice(TagValue: variant; UserName, DiskName: string; CB: TQTXUnRegisterLocalDeviceCB); begin WriteToLogF("Removing local device [%s] for user [%s] ", [DiskName, Username]); // Check username string UserName := username.trim().ToLower(); if Username.length < 1 then begin WriteToLog("Failed to unregister device, username was invalid error"); var lError := EException.Create("Failed to register device, invalid username"); if assigned(CB) then CB(TagValue, DiskName, lError) else raise lError; exit; end; // Check diskname string DiskName := DiskName.trim().ToLower(); if DiskName.length < 1 then begin WriteToLog("Failed to unregister device, disk-name was invalid error"); var lError := EException.Create("Failed to register device, invalid disk-name"); if assigned(CB) then CB(TagValue, DiskName, lError) else raise lError; exit; end; FindDeviceByName(TagValue, Username, DiskName, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin // Did the search fail? if Error nil then begin WriteToLog(Error.message); if assigned(CB) then CB(TagValue, DiskName, Error) else raise Error; exit; end; // Make sure the device is local if Device.&Type dtLocal then begin var lError := EException.CreateFmt('Failed to translate path, device [%s] is not local error', [Device.Name]); if assigned(CB) then CB(TagValue, DiskName, Error) else raise Error; exit; end; // Delete record from database var lWriter: TQTXWriteTransaction; if FDatabase.CreateWriteTransaction(lWriter) then begin lWriter.SQL := "delete from profiles where user = ? and name = ?;"; lWriter.Parameters.AddValueOnly(Username); lWriter.Parameters.AddValueOnly(DiskName); lWriter.Execute( procedure (Sender: TObject; Error: Exception) begin try if Error nil then begin if assigned(CB) then CB(TagValue, DiskName, Error) else raise Error; exit; end; // Construct complete storage location var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices'; lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim(); // Now delete the disk-drive directory TQTXNodeFileUtils.DeleteDirectory(nil, lFullPath, procedure (TagValue: variant; Path: string; Error: Exception) begin if assigned(CB) then CB(TagValue, DiskName, Error) end); finally lWriter.free; lWriter := nil; end; end); end; end); end; procedure TQTXTaskService.RegisterLocalDevice(TagValue: variant; Username, DiskName: string; CB: TQTXRegisterLocalDeviceCB); begin WriteToLogF("Adding local device [%s] for user [%s] ", [DiskName, Username]); UserName := username.trim().ToLower(); if Username.length < 1 then begin WriteToLog("Failed to register device, username was invalid error"); var lError := EException.Create("Failed to register device, invalid username"); if assigned(CB) then CB(TagValue, '', lError) else raise lError; exit; end; DiskName := DiskName.trim().ToLower(); if DiskName.length < 1 then begin WriteToLog("Failed to register device, disk-name was invalid error"); var lError := EException.Create("Failed to register device, invalid disk-name"); if assigned(CB) then CB(TagValue, '', lError) else raise lError; exit; end; FindDeviceByName(TagValue, Username, DiskName, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin // Did the search fail? if Error nil then begin WriteToLog(Error.message); if assigned(CB) then CB(TagValue, '', Error) else raise Error; exit; end; // Does a device that match already exist? if Device nil then begin var lError := EException.CreateFmt("Failed to create device [%s], device already exists", [DiskName]); if assigned(CB) then CB(TagValue, '', lError) else raise lError; exit; end; // make sure the device-folder does not exist, so we can create it ValidateLocalDiskName(TagValue, Username, DiskName, procedure (TagValue: variant; PathName: string; Error: Exception) begin if Error nil then begin if assigned(CB) then CB(TagValue, '', Error) else raise Error; exit; end; // ValidateLocalDiskName only returns the valid directory-name, // not a full path -- so we need to build up the full targetpath var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices'; lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + PathName; TQTXNodeFileUtils.CreateDirectory(nil, lFullPath, procedure (TagValue: variant; Path: string; Error: exception) begin if Error nil then begin var lError := EException.CreateFmt("Failed to create device [%s] with path: %s", [DiskName, lFullPath]); if assigned(CB) then CB(TagValue, PathName, lError) else raise lError; exit; end; FDatabase.Execute( #'insert into devices (type, owner, name, location) values(?, ?, ?, ?);', [ord(JDeviceType.dtLocal), UserName, Diskname, PathName] , procedure (Sender: TObject; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); if assigned(CB) then CB(TagValue, PathName, Error) else raise Error; exit; end; WriteToLogF("Device [%s] added to database user [%s]", [DiskName, UserName]); if assigned(CB) then CB(TagValue, PathName, nil); end); end); end); end); end; procedure TQTXTaskService.SetupDeviceTable(const TagValue: variant; const CB: TRagnarokServiceCB); begin FDatabase.Execute( #' create table if not exists devices ( id integer primary key AUTOINCREMENT, type integer, owner text, name text, location text, apikey text, apisecret text, apipassword text, apiuser text ); ', [], procedure (Sender: TObject; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); if assigned(CB) then CB(Error) else raise Error; exit; end else if assigned(CB) then CB(nil); end); end; procedure TQTXTaskService.SetupTaskTable(const TagValue: variant; const CB: TRagnarokServiceCB); begin FDatabase.Execute( #' create table if not exists tasks ( id integer primary key AUTOINCREMENT, state integer, username text, created real ); ', [], procedure (Sender: TObject; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); if assigned(CB) then CB(Error) else raise Error; exit; end else if assigned(CB) then CB(nil); end); end; procedure TQTXTaskService.SetupOperationsTable(const TagValue: variant; const CB: TRagnarokServiceCB); begin FDatabase.Execute( #' create table if not exists operations ( id integer primary key AUTOINCREMENT, username text, taskid integer, name text, filename text ); ', [], procedure (Sender: TObject; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); if assigned(CB) then CB(Error) else raise Error; exit; end else if assigned(CB) then CB(nil); end); end; procedure TQTXTaskService.SetupDatabase(const CB: TRagnarokServiceCB); begin // Try to read database-path from preferences file var LDbFileToOpen := FPrefs.ReadString("database", "database_name", ""); // Trim away spaces, check if there is a filename LDbFileToOpen := LDbFileToOpen.trim(); if LDbFileToOpen.length < 1 then begin // No filename? Fall back on pre-defined file in CWD var LBasePath := TQTXNodeFileUtils.GetCurrentDirectory(); LDbFileToOpen := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(LBasePath) + CNT_PREFS_DBNAME; end; FDatabase.AccessMode := TSQLite3AccessMode.sqaReadWriteCreate; FDatabase.Open(LDbFileToOpen, procedure (Sender: TObject; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); if assigned(CB) then CB(Error) else raise Error; exit; end; WriteToLog("Initializing task table"); SetupTaskTable(nil, procedure (Error: exception) begin if Error nil then begin WriteToLog("Tasks initialized: **failed"); WriteToLog(error.message); if assigned(CB) then CB(Error) else raise Error; exit; end else writeToLog("Tasks initialized: OK"); WriteToLog("Initializing operations table"); SetupOperationsTable(nil, procedure (Error: exception) begin if Error nil then begin WriteToLog("Operations initialized: **failed"); WriteToLog(error.message); if assigned(CB) then CB(Error); exit; end else writeToLog("Operations initialized: OK"); WriteToLog("Initializing device table"); SetupDeviceTable(nil, procedure (Error: exception) begin if Error nil then begin WriteToLog("Device-table initialized: **failed"); WriteToLog(error.message); if assigned(CB) then CB(Error); exit; end else writeToLog("Device-table initialized: OK"); if assigned(CB) then CB(nil); end); end); end); end); end; procedure TQTXTaskService.HandleFileRead(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileReadRequest(request); var lUserName := lRequest.UserName; var lFileName := lRequest.FileName; // Check filename length if lFileName.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; GetFileInfo(lRequest, lUserName, lFileName, procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; var lOptions: TReadFileOptions; lOptions.encoding := 'binary'; NodeJsFsAPI().readFile(LocalFile, lOptions, procedure (Error: JError; Data: JNodeBuffer) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; var lResponse := TQTXFileReadResponse.Create(Request.Ticket); lResponse.UserName := lUserName; lResponse.Routing.TagValue := request.routing.tagValue; lResponse.FileName := lFileName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; // Convert filedata in one pass try var lConvert := TDataTypeConverter.Create(); try lResponse.Attachment.AppendBytes( lConvert.TypedArrayToBytes(Data) ); finally lConvert.free; end; except on e: exception do begin WriteToLog(e.message); SendError(Socket, Request, e.Message); exit; end; end; try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end); end; procedure TQTXTaskService.HandleFileReadPartial(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileReadPartialRequest(request); var lUserName := lRequest.UserName; var lFileName := lRequest.FileName; var lStart := lRequest.Offset; var lSize := lRequest.Size; // Check filename length if lFileName.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; if lSize < 1 then begin SendError(Socket, Request, "Read failed, invalid size error"); exit; end; if lStart < 0 then begin SendError(Socket, Request, "Read failed, invalid offset error"); exit; end; GetFileInfo(lRequest, lUserName, lFileName, procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; if lStart > Info.size then begin SendError(Socket, Request, "Read failed, offset beyond filesize error"); exit; end; NodeJsFsAPI().open(LocalFile, "r", procedure (Error: JError; Fd: THandle) begin if error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; var Data = new JNodeBuffer(lSize); NodeJsFsAPI().read(Fd, Data, 0, lSize, lStart, procedure (Error: JError; BytesRead: integer; buffer: JNodeBuffer) begin if Error nil then begin NodeJsFsAPI().closeSync(Fd); WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; // Close the file-handle and return data NodeJsFsAPI().close(Fd, procedure (Error: JError) begin var lResponse := TQTXFileReadPartialResponse.Create(Request.Ticket); lResponse.UserName := lUserName; lResponse.Routing.TagValue := request.routing.tagValue; lResponse.FileName := lFileName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; // Only encode data if read if BytesRead > 0 then begin // Convert filedata in one pass try var lConvert := TDataTypeConverter.Create(); try lResponse.Attachment.AppendBytes( lConvert.TypedArrayToBytes(buffer) ); finally lConvert.free; end; except on e: exception do begin WriteToLog(e.message); SendError(Socket, Request, e.Message); exit; end; end; end; try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end); end); end); end; procedure TQTXTaskService.HandleFileWrite(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileWriteRequest(request); var lFileName := lRequest.FileName.trim(); var lUserName := lRequest.UserName.trim(); var FullPath := lFileName; // Check filename length if lFileName.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; var lParser := TQTXPathParser.Create(); try var lInfo: TQTXPathData; if lparser.Parse(FullPath, lInfo) then begin // Locate the device for the path belonging to the user FindDeviceByName(nil, lUserName, lInfo.MountPart, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; case Device.&Type of dtLocal: begin // We want the path + filename, so we can append that to // the actual localized filesystem var lExtract := FullPath; delete(lExtract, 1, lInfo.MountPart.Length + 1); // Construct complete storage location var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices'; lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + lExtract; // Extract data to be appended, if any // note: null bytes should be allowed, it should just create the file var lBytes: array of UInt8; if lRequest.attachment.Size > 0 then lBytes := lRequest.Attachment.ToBytes(); // Write the data to the file NodeJsFsAPI().writeFile(lFullPath, lBytes, procedure (Error: JError) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; // Setup response object var lResponse := TQTXFileWriteResponse.Create(lRequest.Ticket); lResponse.UserName := lUserName; lResponse.FileName := lFileName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; // Send success response try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end; dtDropbox, dtGoogle, dtMsDrive: begin var lErrorText := Format("Clound bindings not active error [%s]", [lRequest.FileName]); WriteToLog(lErrorText); SendError(Socket, Request, lErrorText); end; end; end); end else begin SendError(Socket, Request, format("Invalid path [%s] error", [FullPath])); end; finally lParser.free; end; end; procedure TQTXTaskService.HandleFileWritePartial(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileWritePartialRequest(request); var lFileName := lRequest.FileName.trim(); var lUserName := lRequest.UserName.trim(); var lFileOffset := lRequest.Offset; // Check filename length if lFileName.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; var FullPath := lFileName; var lParser := TQTXPathParser.Create(); try var lInfo: TQTXPathData; if lparser.Parse(FullPath, lInfo) then begin // Locate the device for the path belonging to the user FindDeviceByName(nil, lUserName, lInfo.MountPart, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; case Device.&Type of dtLocal: begin // We want the path + filename, so we can append that to // the actual localized filesystem var lExtract := FullPath; delete(lExtract, 1, lInfo.MountPart.Length + 1); // Construct complete storage location var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices'; lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim(); lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + lExtract; // Extract data to be appended, if any // note: null bytes should be allowed, it should just create the file var lBytes: array of UInt8; if lRequest.attachment.Size > 0 then lBytes := lRequest.Attachment.ToBytes(); var lAccess := TQTXNodeFile.Create(); lAccess.Open(lFullPath, TQTXNodeFileMode.nfWrite, procedure (Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; lAccess.Write(lBytes, lFileOffset, procedure (Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; // Setup response object var lResponse := TQTXFileWriteResponse.Create(lRequest.Ticket); lResponse.UserName := lUserName; lResponse.FileName := lFileName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; // Send success response try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end); end; dtDropbox, dtGoogle, dtMsDrive: begin var lErrorText := Format("Clound bindings not active error [%s]", [lRequest.FileName]); WriteToLog(lErrorText); SendError(Socket, Request, lErrorText); end; end; end); end else begin SendError(Socket, Request, format("Invalid path [%s] error", [FullPath])); end; finally lParser.free; end; end; procedure TQTXTaskService.HandleRmDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXRmDirRequest(request); var lUserName := lRequest.UserName.trim(); var lDirPath := lRequest.DirPath.trim(); if lDirPath.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lDirPath) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; var lParser := TQTXPathParser.Create(); try var lInfo: TQTXPathData; if lParser.Parse(lDirPath, lInfo) then begin GetTranslatedPathFor(nil, lUserName, lDirPath, procedure (TagValue: variant; Original, Translated: string; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; if not TQTXNodeFileUtils.DirectoryExists(Translated) then begin WriteToLogF("RmDir Failed, directory [%s] does not exist", [Translated]); SendError(Socket, Request, Format("RmDir failed, directory [%s] does not exist", [Original])); exit; end; TQTXNodeFileUtils.DeleteDirectory(nil, Translated, procedure (TagValue: variant; Path: string; Error: Exception) begin if error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; // Setup response object var lResponse := TQTXRmDirResponse.Create(lRequest.Ticket); lResponse.UserName := lUserName; lResponse.DirPath := lDirPath; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; lResponse.Routing.TagValue := lRequest.Routing.TagValue; // Send success response try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end); end else begin var lText := format("RmDir failed, invalid path [%s] error", [lDirPath]); WriteToLog(lText); SendError(Socket, Request, lText); end; finally lParser.free; end; end; procedure TQTXTaskService.HandleMkDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXMkDirRequest(request); var lUserName := lRequest.UserName.trim(); var lDirPath := lRequest.DirPath.trim(); if lDirPath.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lDirPath) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; var lParser := TQTXPathParser.Create(); try var lInfo: TQTXPathData; if lparser.Parse(lDirPath, lInfo) then begin GetTranslatedPathFor(nil, lUserName, lDirPath, procedure (TagValue: variant; Original, Translated: string; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; TQTXNodeFileUtils.DirectoryExists(nil, Translated, procedure (TagValue: variant; Path: string; Error: Exception) begin if Error nil then begin WriteToLogF("MkDir Failed, directory [%s] already exists", [Translated]); SendError(Socket, Request, Format("MkDir Failed, directory [%s] already exists", [Original])); exit; end; TQTXNodeFileUtils.CreateDirectory(nil, Translated, procedure (TagValue: variant; Path: string; Error: Exception) begin if Error nil then begin WriteToLogF("MkDir Failed, directory [%s] could not be created", [Original]); SendError(Socket, Request, Format("MkDir Failed, directory [%s] could not be created", [Translated])); exit; end; // Setup response object var lResponse := TQTXMkDirResponse.Create(lRequest.Ticket); lResponse.UserName := lUserName; lResponse.DirPath := lDirPath; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; lResponse.Routing.TagValue := lRequest.Routing.TagValue; // Send success response try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end); end); end else begin var lText := format("MkDir Failed, invalid path [%s] error", [lDirPath]); WriteToLog(lText); SendError(Socket, Request, lText); end; finally lParser.free; end; end; procedure TQTXTaskService.HandleFileDelete(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileDeleteRequest(Request); var lUserName := lRequest.UserName.trim(); var lFileName := lRequest.FileName.trim(); if lFileName.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; GetFileInfo(lRequest, lUserName, lFileName, procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; if not Info.isFile then begin SendError(Socket, Request, "Filesystem object is not a file error"); exit; end; NodeJsFsAPI().unlink(LocalFile, procedure (Error: JError) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.message); exit; end; var lResponse := new TQTXFileDeleteResponse(lRequest.Ticket); lResponse.Routing.TagValue := request.Routing.TagValue; lResponse.UserName := lUserName; lResponse.FileName := lFileName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end); end; procedure TQTXTaskService.HandleFileRename(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileRenameRequest(Request); var lUserName := lRequest.UserName.trim(); var lFileName := lRequest.FileName.trim(); var lNewName := lRequest.NewName.trim(); // Check filename length if lFileName.length < 1 then begin SendError(Socket, Request, Format("Invalid or empty from-filename [%s] error", [lFileName]) ); exit; end; // check newname length if lNewName.length 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; if pos(lTemp, lNewName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; if pos(lTemp, lNewName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; GetFileInfo(lRequest, lUserName, lFileName, procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; if not Info.isFile then begin SendError(Socket, Request, "Filesystem object is not a file error"); exit; end; GetTranslatedPathFor(nil, lUsername, lNewName, procedure (TagValue: variant; Original, Translated: string; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; NodeJsFsAPI().rename(LocalFile, Translated, procedure (Error: JError) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.message); exit; end; var lResponse := new TQTXFileRenameResponse(lRequest.Ticket); lResponse.Routing.TagValue := request.Routing.TagValue; lResponse.UserName := lUserName; lResponse.FileName := lFileName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end); end); end; procedure TQTXTaskService.HandleGetDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileDirRequest(Request); var lUserName := lRequest.UserName.trim(); var lPath := lRequest.Path.trim(); // prevent path escape attempts var lTemp := "../"; if pos(lTemp, lPath) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lPath) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; GetTranslatedPathFor(nil, lUserName, lPath, procedure (TagValue: variant; Original, Translated: string; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; //writeln("Translated path is:" + Translated); if not TQTXNodeFileUtils.DirectoryExists(Translated) then begin WriteToLogF("GetDir Failed, directory [%s] does not exist", [Translated]); SendError(Socket, Request, Format("GetDir failed, directory [%s] does not exist", [Original])); exit; end; var lWalker := TQTXFileWalker.Create(); lWalker.Examine(Translated, procedure (Sender: TQTXFileWalker; Error: EException) begin if Error nil then begin WriteToLogF("GetDir Failed: %s", [Error.Message]); SendError(Socket, Request, Format("GetDir failed: %s", [Error.Message])); exit; end; // Get the directory data, swap out the path // record with the original [amiga] style path var lData := Sender.ExtractList(); lData.dlPath := Original; var lResponse := new TQTXFileDirResponse(lRequest.Ticket); lResponse.Routing.TagValue := request.Routing.TagValue; lResponse.UserName := lUserName; lResponse.Path := lPath; lResponse.Assign( lData ); try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; // release instance in 100ms TQTXDispatch.execute(procedure () begin try lWalker.free except on e: exception do begin WriteToLogF("Failed to release file-walker instance: %s", [e.message]); end; end; end, 100); end); end); end; procedure TQTXTaskService.HandleGetFileInfo(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lRequest := TQTXFileInfoRequest(Request); var lUserName := lRequest.UserName.trim(); var lFileName := lRequest.FileName.trim(); // prevent path escape attempts var lTemp := "../"; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; lTemp := './'; if pos(lTemp, lFileName) > 0 then begin SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) ); exit; end; GetFileInfo(lRequest, lUserName, lFileName, procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception) begin if Error nil then begin WriteToLog(Error.message); SendError(Socket, Request, Error.Message); exit; end; // Collect the data var lData := new JFileItem(); lData.diFileName := lFileName; lData.diFileType := if Info.isFile then JFileItemType.wtFile else JFileItemType.wtFolder; lData.diFileSize := Info.size; lData.diFileMode := IntToStr(Info.mode); lData.diCreated := TDateUtils.FromJsDate( Info.cTime ); lData.diModified := TDateUtils.FromJsDate( Info.mTime ); var lResponse := new TQTXFileInfoResponse(lRequest.Ticket); lResponse.Routing.TagValue := request.Routing.TagValue; lResponse.UserName := lUserName; lResponse.FileName := lFileName; lResponse.Assign(lData); try Socket.Send( lResponse.Serialize() ); except on e: exception do WriteToLog(e.message); end; end); end; procedure TQTXTaskService.HandleDestroyDevice(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lMessage := TQTXFileDestroyDeviceRequest(request); // This will also destroy any files + unregister the device in the // database table for the service -- do not mess with this! UnRegisterLocalDevice(nil, lMessage.Username, lMessage.DeviceName, procedure (TagValue: variant; LocalPath: string; Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; var lResponse := TQTXFileDestroyDeviceResponse.Create(request.ticket); lResponse.UserName := lMessage.UserName; lResponse.DeviceName := lMessage.DeviceName; lResponse.Routing.TagValue := Request.Routing.TagValue; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; try Socket.Send( lResponse.Serialize() ); except on e: exception do begin WriteToLog(e.message); end; end; end); end; procedure TQTXTaskService.HandleCreateLocalDevice(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lMessage := TQTXFileCreateLocalDeviceRequest(request); // Attempt to register. // NOTE: This will automatically create a matching folder // under $cwd/userdevices/[calculated_name_of_device] RegisterLocalDevice(nil, lMessage.Username, lMessage.DeviceName, procedure (TagValue: variant; LocalPath: string; Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; FindDeviceByName(nil, lMessage.Username, lMessage.DeviceName, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; var lResponse := TQTXFileCreateLocalDeviceResponse.Create(request.ticket); lResponse.UserName := lMessage.UserName; lResponse.Routing.TagValue := Request.Routing.TagValue; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; if Device nil then lResponse.assign(Device); try Socket.Send( lResponse.Serialize() ); except on e: exception do begin WriteToLog(e.message); end; end; end); end); end; procedure TQTXTaskService.HandleGetDeviceByName(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lMessage := TQTXFileGetDeviceByNameRequest(request); FindDeviceByName(nil, lMessage.Username, lMessage.DeviceName, procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; var lResponse := TQTXFileGetDeviceByNameResponse.Create(request.ticket); lResponse.UserName := lMessage.UserName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; if Device nil then lResponse.assign(Device); try Socket.Send( lResponse.Serialize() ); except on e: exception do begin WriteToLog(e.message); end; end; end); end; procedure TQTXTaskService.HandleGetDevices(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage); begin var lMessage := TQTXFileGetDeviceListRequest(Request); GetDevicesForUser(nil, lMessage.Username, procedure (TagValue: variant; Devices: JDeviceList; Error: Exception) begin if Error nil then begin WriteToLog(Error.Message); SendError(Socket, Request, Error.Message); exit; end; var lResponse := TQTXFileGetDeviceListResponse.Create(request.ticket); lResponse.UserName := lMessage.UserName; lResponse.Code := CNT_MESSAGE_CODE_OK; lResponse.Response := CNT_MESSAGE_TEXT_OK; if Devices nil then lResponse.assign(Devices); try Socket.Send( lResponse.Serialize() ); except on e: exception do begin WriteToLog(e.message); end; end; end); end; procedure TQTXTaskService.AfterServerStarted; begin inherited; // Check prefs if zconfig should be applied if self.FPrefs.ReadBoolean("zconfig", "active", false) then begin // ZConfig should only run on the master instance. // We dont want to register our endpoint for each worker if NodeJSClusterAPI().isWorker then exit; writeln("Setting up Zero-Configuration layer"); FZConfig.port := FPrefs.ReadInteger('zconfig', 'bindport', 2109); FZConfig.address := GetMachineIP(); FZConfig.Start(nil, procedure (Sender: TObject; TagValue: variant; Error: Exception) begin if FPrefs.ReadBoolean("zconfig", "broadcast", true) then FZConfig.Socket.setBroadcast(true); // Build up the endpoint (URL) for our websocket server var lEndpoint := ''; if FPrefs.ReadBoolean('networking', 'secure', false) then lEndpoint := 'wss://' else lEndpoint := 'ws://'; lEndpoint += GetMachineIP(); lEndpoint += ':' + Port.ToString(); // Ping the ZConfig service on interval, until our service is registered // We keep track of the interval handle so we can stop calling on interval later FRegHandle := TQTXDispatch.SetInterval( procedure () begin inc(FRegCount); // Only output once to avoid overkill in the log if FRegCount = 1 then WriteToLogF("ZConfig registration begins [%s]", [lEndpoint]); FZConfig.RegisterService(nil, CNT_ZCONFIG_SERVICE_NAME, SERVICE_ID_TASKMANAGER, lEndpoint, procedure (TagValue: variant; Error: Exception) begin if Error = nil then begin WriteToLog("Service registered"); TQTXDispatch.ClearInterval(FRegHandle); FRegCount := 0; exit; end; end); end, 1000); end); end; end; procedure TQTXTaskService.BeforeServerStopped; begin inherited; end; procedure TQTXTaskService.Dispatch(Socket: TNJWebSocketSocket; Message: TQTXBaseMessage); begin var LInfo := MessageDispatch.GetMessageInfoForClass(Message); if LInfo nil then begin try LInfo.MessageHandler(Socket, Message); except on e: exception do begin //Log error WriteToLog(e.message); end; end; end; end; end.
Quartex Media Desktop, new compiler and general progress
It’s been a few weeks since my last update on the project. The reason I dont blog that often about Quartex Media Desktop (QTXMD), is because the official user-group has grown to 2000+ members. So it’s easier for me to post developer updates directly to the audience rather than writing articles about it.
If you haven’t bothered digging into the project, let me try to sum it up for you quickly.
Quick recap on Quartex Media Desktop
To understand what makes this project special, first consider the relationship between Microsoft Windows and a desktop program. The operating system, be it Windows, Linux or OSX – provides an infrastructure that makes complex applications possible. The operating-system offers functions and services that programs can rely on.
The most obvious being:
- A filesystem and the ability to save and load data
- A windowing toolkit so programs can be displayed and have a UI
- A message system so programs can communicate with the OS
- A service stack that takes care of background tasks
- Authorization and identity management (security)
I have just described what the Quartex Media Desktop is all about. The goal is simple:
to provide for JavaScript what Windows and OS X provides for ordinary programs.
Just stop and think about this. Every “web application” you have ever seen, have all lacked these fundamental features. Sure you have libraries that gives you a windowing environment for Javascript, like Embarcadero Sencha; but im talking about something a bit more elaborate. Creating windows and buttons is easy, but what about ownership? A runtime environment has to keep track of the resources a program allocates, and make sure that security applies at every step.
Target audience and purpose
Take a second and think about how many services you use that have a web interface. In your house you probably have a router, and all routers can be administered via the browser. Sadly, most routers operate with a crude design and that leaves much to be desired.
If you like to watch movies you probably have a Plex or Kodi system running somewhere in your house; perhaps you access that directly via your TV – or via a modern media system like Playstation 4 or XBox one. Both Plex and Kodi have web-based interfaces.
Netflix is now omnipresent and have practically become an institution in it’s own right. Netflix is often installed as an app – but the app is just a thin wrapper around a web-interface. That way they dont have to code apps for every possible device and OS out there.
If you commute via train in Scandinavia, chances are you buy tickets on a kiosk booth. Most of these booths run embedded software and the interface is again web based. That way they can update the whole interface without manually installing new software on each device.
These are just examples of web based interfaces you might know and use; devices that leverage web technology. As a developer, wouldn’t it be cool if there was a system that could be forked, adapted and provide advanced functionality out of the box?
Just imagine a cheap Jensen router with a Quartex Media Desktop interface! It could provide a proper UI interface with applications that run in a windowing environment. They could disable ordinary desktop functionality and run their single application in kiosk mode. Taking full advantage of the underlying functionality without loss of security.
And the same is true for you. If you have a great idea for a web based application, you can fork the system, adjust it to suit your needs – and deploy a cutting edge cloud system in days rather than months!
New compiler?
Up until recently I used Smart Mobile Studio. But since I have left that company, the matter became somewhat pressing. I mean, QTXMD is an open-source system and cant really rely on third-party intellectual property. Eventually I fired up Delphi, forked the latest DWScript, and used that to roll a new command-line compiler.
But with a new compiler I also need a new RTL. Thankfully I have been coding away on the new RTL for over a year, but there is still a lot of work to do. I essentially have to implement the same functionality from scratch.
There will be more info on the new compiler / codegen when its production ready.
Progress
If I was to list all the work I have done since my last post, this article would be a small book. But to sum up the good stuff:
- Authentication has been moved into it’s own service
- The core (the main server) now delegates login messages to said service
- We no longer rely on the Smart Pascal filesystem drivers, but use the raw node.js functions instead (faster)
- The desktop now use the Smart Theme engine. This means that we can style the desktop to whatever we like. The OS4 theme that was hardcoded will be moved into its own proper theme-file. This means the user can select between OS4, iOS, Android and Ubuntu styling. Creating your own theme-files is also possible. The Smart theme-engine will be replaced by a more elaborate system in QTX later
- Ragnarok (the message api) messages now supports routing. If a routing structure is provided, the core will relay the message to the process in question (providing security allows said routing for the user)
- The desktop now checks for .info files when listing a directory. If a file is accompanied by an .info file, the icon is extracted and shown for that file
- Most of the service layer now relies on the QTX RTL files. We still have some dependencies on the Smart Pascal RTL, but we are making good progress on QTX. Eventually the whole system will have no dependencies outside QTX – and can thus be compiled without any financial obligations.
- QTX has it’s own node.js classes, including server and client base-classes
- Http(s) client and server classes are added to QTX
- Websocket and WebSocket-Secure are added to QTX
- TQTXHybridServer unifies http and websocket. Meaning that this server type can handle both orinary http requests – but also websocket connections on the same network socket. This is highly efficient for websocket based services
- UDP classes for node.js are implemented, both client and server
- Zero-Config classes are now added. This is used by the core for service discovery. Meaning that the child services hosted on another machine will automatically locate the core without knowing the IP. This is very important for machine clustering (optional, you can define a clear IP in the core preferences file)
- Fixed a bug where the scrollbars would corrupt widget states
- Added API functions for setting the scrollbars from hosted applications (so applications can tell the desktop that it needs scrollbar, and set the values)
- .. and much, much more
I will keep you all posted about the progress — the core (the fundamental system) is set for release in december – so time is of the essence! Im allocating more or less all my free time to this, and it will be ready to rock around xmas.
When the core is out, I can focus solely on the applications. Everything from Notepad to Calculator needs to be there, and more importantly — the developer tools. The CloudForge IDE for developers is set for 2020. With that in place you can write applications for iOS, Android, Windows, OS X and Linux directly from Quartex Media Desktop. Nothing to install, you just need a modern browser and a QTX account.
The system is brilliant for small teams and companies. They can setup their own instance, communicate directly via the server (text chat and video chat is scheduled) and work on their products in concert.
Repository updates
As most know by now, I was running a successful campaign on Patreon until recently. I know that some are happy with Patreon, but hopefully my experience will be a wakeup call about the total lack of rights you as a creator have – should Patreon decide they don’t understand what you are doing (which I can only presume was the case, because I was never given a reason at all). You can read more about my experience with Patreon by clicking here.
Setting up repositories
Having to manually build a package for each tier that I have backers for would be a disaster. It was time-consuming and repetitive enough to create packages on Patreon, and I don’t have time to reverse engineer Patreon either. Which I might do in the future and release as open-source just to give them a kick in the groin back.
To make it easier for my backers to get the code they want, I have isolated each project and sub-project in separate repositories on BitBucket. This covers Delphi, Smart Pascal, LDEF and everything else.
I’m just going to continue with the Tiers I originally made on Patreon, and use my blog as the news-center for everything. Since I tend to blog about things from a personal point of view, be it for Delphi, JavaScript or Smart Pascal — I doubt people will notice the difference.
So far the following repositories have been setup:
- Amibian.js Server (Quartex Web OS)
- Amibian.js Client
- HexLicense
- TextCraft (source-code parser for Delphi and Smart Pascal)
- UAE.js (a fork of SAE, the JS implementation of UAE)
I need to clean up the server repository a bit, because right now it contains both the server-code and various sub projects. The LDEF assembler program for example, is also under that repository — and it belongs in its own repository as a unique sub-project.
The following repositories will be setup shortly:
- Tweening library for Delphi and Smart Pascal
- PixelRage graphics library
- ByteRage bugger library
- LDEF (containing both Delphi and Smart Pascal code)
- LDEF Assembler
It’s been extremely busy days lately so I need to do some thinking about how we can best organize things. But rest assured that everyone that backs the project, or a particular tier, will get access to what they support.
Support and backing
I have been looking at various ways to do this, but since most backers have just said they want Paypal, I decided to go for that. So donations can be done directly via paypal. One of the new features in Paypal is repeated payments, so setting up a backer-plan should be easy enough. I am notified whenever someone gives a donation, so it’s pretty easy to follow-up on.
Updates used to be monthly, but with the changes they will be ad-hoc, meaning that I will commit directly. I do have local backups and a local git server, so for parts of the project the commits will be issued at the end of each month.
While all support is awesome, here are the tiers I used on Patreon:
- $5 – “high-five”, im not a coder but I support the cause
- $10 – Tweening animation library
- $25 – License management and serial minting components
- $35 – Rage libraries: 2 libraries for fast graphics and memory management
- $45 – LDef assembler, virtual machine and debugger
- $50 – Amibian.js (pre compiled) and Ragnarok client / server library
- $100 – Amibian.js binaries, source and setup
- $100+ All the above and pre-made disk images for ODroid XU4 and x86 on completion of the Amibian.js project (12 month timeline).
So to back the project like before, all you do is:
- Register with Bitbucket (free user account)
- Setup donation and inform me of your Bitbucket user-name
- I add you on BitBucket so you are granted access rights
Easy. Fast and reliable.
The QTX RTL
Those that have been following the Amibian.js project might have noticed that a fair bit of QTX units have appeared in the code? QTX is a run-time library compatible with Smart Mobile Studio and DWScript. Eventually the code that makes up Amibian.js will become a whole new RTL. This RTL has nothing to do with Smart Mobile Studio and ships with its own license.
Backers at $45 or beyond access to this code automatically. If you use Smart Mobile Studio then this is a must. It introduces a ton of classes that doesn’t exist in Smart Pascal, and also introduces a much faster and clean visual component framework.
If you want to develop visual applications using QTX and DWScript, then that is OK, providing the license is respected (LGPL, non commercial use).
Well, stay tuned for more info and news!
Mirroring groups on the MeWe network
Following my Administrator woes on Facebook post I have had a look at alternative places to run a forum. I realized that Facebook is getting pretty intrinsic in society around the world, so I know everyone won’t be interested in a new venue. But honestly, MeWe is very simple to use and have an UI experience very close to the Facebook app.
MeWe is thankfully not like G+ which (in my personal opinion) was counter-intuitive and damn right intrusive. We all remember the G+ auto-upload feature, where some 3 million users had their family photos, vacation photos and .. ehrm, “explicitly personal” photos uploaded without consent.
Well, the MeWe app is very simple, and registration is as easy as it should be. You make a user name, a password, and type in your email; then you verify your email and that’s it!
Besides, my main use for Facebook or MeWe is to run the groups – I spend very little of my time socializing anyways. With the amount of groups and media i push on a daily basis it’s quite frankly their loss.
The alternative to MeWe is to setup a proper web forum instead. I have bought 6 domains that are now collecting dust so yes, I will look into that – but the whole purpose of a social platform is that you don’t have to do maintenance beyond daily management – so MeWe saves us some time.
So head over to MeWe and register! Here are the two main groups I manage these days. The main groups are on facebook, but i have now registered the same groups on MeWe.
MeWe doesn’t cost anything and takes less than 5 minutes to join. Just like G+ and Facebook, MeWe can be installed as an app for your phone (both iOS and Android). So as far as alternatives go, it’s a good alternative. One more app wont do much harm I imagine.
Note: I will naturally keep my Facebook account for the sake of the groups, but having experienced this 4 times in 9 years, my tolerance of Mr. Suckerberg is quickly reaching its limits. If I have blurted something out I have no problems standing for that and taking the penalty, but posting a picture of software development? In a group dedicated to software development? That takes some impressive mental acrobatics to accept.
The Amiga ARM project
This has been quite the turbulent week. Without getting into all the details, a post that I made with thoughts and ideas for an Amiga inspired OS for ARM escaped the safe confines of our group, Amiga Disrupt, and took on a life of its own.
This led to a few critical posts being issued publicly, which all boiled down to a misunderstanding. Thankfully this has been resolved and things are back to normal.
The question on everyone’s lips now seem to be: did Jon mean what he said or was it just venting frustration? I thought I made my points clear in my previous post, but sadly Commodore USA formulated a title open for interpretation (which is understandable considering the mayhem at the time). So let’s go thrugh the ropes and put this to rest.
Am I making an ARM based Amiga inspired OS?
Hopefully I don’t have to. My initial post, the one posted to the Amiga Disrupt comment section (and mistaken for a project release note), had a couple of very clear criteria attached:
If nothing has been done to improve the Amiga situation [with regards to ARM or x86] by the time I finish Amibian.js (*), I will take matters into my own hand and create my own alternative.
(*) As you probably know, Amibian.js is a cloud implementation of Amiga OS, designed to bring Amiga to the browser. It is powered by a node.js application server; a server that can be hosted either locally (on the same machine as the html5 client) or remotely. It runs fine on popular embedded devices such as Tinkerboard and ODroid, and when run in a full-screen browser with no X or Windows desktop behind it – it is practically indistinguishable from the real thing.
We have customers who use our prototype to deliver cloud based learning for educational institutions. Shipping ready to use hardware units with pre-baked Amibian.js installed is perfect for schools, libraries, museums, routers and various kiosk projects.
Note: This project started years before FriendOS, so we are not a clone of their work.
Obviously this is a large task for one person, but I have written the whole system in Smart Mobile Studio, which is a product our company started some 7 years ago, and that now has a team of six people behind it. In short it takes object pascal code such as Delphi and Freepascal, and compiles this to JavaScript. Suitable for both the browser and NodeJS. It gives you a full IDE with form designer, drag & drop visual components and a wast and rich RTL (run-time library) which naturally saves me a lot of time. So this gives me an edge over other companies working with similar technology. So while it’s a huge task, it’s leveraged considerably by the toolchain I made for it.
So am I making a native OS for ARM or x86? The short answer: I will if the situation havent dramatically improved by the time Amibian.js is finished.
Instead of wasting years trying to implement everything from scratch, Pascal Papara took the Linux kernel and ran with it. So Aeros boots by virtue of the Linux Kernel, but jumps straight into Aros once the drivers has loaded
If you are thinking “so what, who the hell do you think you are?” then perhaps you should take a closer look at my work and history.
I am an ex Quartex member, which was one of the most infamous hacking cartels in europe. I have 30 years of software development behind me, having worked as a professional developer since the age of 17. I have a history of taking on “impossible” projects and finding ways to deliver them. Smart Mobile Studio itself was deemed impossible by most Delphi developers; It was close to heresy, triggering an avalanche of criticism for even entertaining the idea that object pascal could be compiled to JavaScript. Let alone thrive on JSVM (JavaScript Virtual Machine).
You can imagine the uproar when our generated JavaScript code (compiled from object pascal) actually bested native code. I must admit we didn’t expect that at all, but it changed the way Delphi and object pascal developers looked at the world – for the better I might add.
What I am good at, is taking ordinary off the shelves parts and assembling them in new and exciting ways. Often ways the original authors never intended; in order to produce something unique. My faith is not in myself, but in the ability and innate capacity of human beings to find solutions. The biggest obstacle to progress is ultimately pride and fear of losing face. Something my Buddhist training beat our of me ages ago.
So this is not an ego trip, it’s simply a coder that is completely fed-up with the perpetual mismanagement that has held Amiga OS in captivity for two decades.
Amiga OS is a formula, and formulas are bulletproof
People love different aspects of the same thing – and the Amiga is no different. For some the Amiga is the games. Others love it for its excellent sound capabilities, while some love it for the ease of coding (the 68k is the most friendly cpu ever invented in my book). And perhaps all of us love the Amiga for the memories we have. A harmless yet valuable nostalgia of better times.
But for me the love was always the OS itself. The architecture of Amiga OS is so elegant and dare I say, pure, compared to other systems. And I’m comparing against both legacy and contemporary systems here. Microsoft Windows (WinAPI) comes close, but the sheer brilliance of Amiga OS is yet to be rivaled.We are talking about a design that delivers a multimedia driven, window based desktop 10 years before the competition. A desktop that would thrive in as little as 512 kb of ram, with fast and reliable pre-emptive multitasking.
I don’t think people realize or understand the true value of Amiga OS. It’s not in the games (although games is definitively a huge part of the experience), the hardware or the programs. The reason people have been fighting bitterly over Amiga OS for a lifetime, is because the operating system architecture or “formula” is unmatched to this very day.
Can you imagine what a system that thrives under 512 KB would do to the desktop market? Or even better, what it could bring to the table for embedded and server technology?
And this is where my frustration soars up. Even though we have OS 4.1, we have been forced to idly stand by and watch, as mistake after mistake is being made. opportunities that are ripe for the taking (some of them literally placed on the doorstep of Hyperion), have been thrown by the wayside time and time again.
And they are not alone. Aros and Morphos has likewise missed a lot of opportunities. Both opportunities to generate income and secure development as well as embracing new technology. Although I must stress that I sympatize with Aros since they lack any official funding. Morphos is doing much better using a normal, commerical license.
Frustration, the mother of invention
When the Raspberry PI was first released I jumped of joy. Finally a SBC (single board computer) with enough power to run a light version of Amiga OS 4.1, with a price tag that everyone can live with. I rushed over to Hyperion to see if they had issued a statement about the PI, but nothing could be found. The AEON site was likewise empty.
The PI version 2 came and went, still no sign that Hyperion would capitalize on the situation. I expected them to issue a “Amiga OS 4.1 light” edition for ARM, which would put them on the map and help them establish a user base. Without a user base and fresh blood there is no chance in hell of selling next generation machines in large enough quantities to justify future development. But once again, opportunity after oppertunity came and went.
Faster and better suited SBC’s started to turn up in droves: The ODroid, Beaglebone black, The Tinkerboard, The Banana PI – and many, many others. When the SnapDragon IV CPU’s shipped on a $120 SBC, which is the same processor used by Samsung Galaxy 6S, I was sure Hyperion would wake up and bring Amiga OS to the masses. But not a word.
Instead we were told to wait for the Amiga x5000 which is based on PPC. I have no problem with PPC, it’s a great platform and packs a serious punch. But since PPC no longer sell to mainstream computer companies like it used to, the price penalty would be nothing short of astronomical. There is also the question of longevity and being able to maintain a PPC based system for the forseeable future. Where exactly is PPC in 15 years?
Note: One of the reasons PPC was selected has to do with coding infrastructure. PPC has an established standard, something ARM lacked at the time (this was first established for ARM in 2014). PPC also has an established set of development platforms that you can build on, with libraries and pre-fab modules (pre fabricated modules, think components that you can use to quickly build what you need) that have been polished for two decades now. A developer who knows PPC from the Amiga days will naturally feel more at home with PPC. But sadly PPC is the past and modern development takes place almost exclusively on ARM and x86. Even x86 is said to have an expiration date now.
The only group that genuinely tried to bring Amiga OS to ARM has been the Aros team. They got their system compiled, implemented some rudimentary drivers (information on this has been thin to say the least) and had it booting natively on the Raspberry PI 3b. Sadly they lacked a USB stack (remember I mentioned pre-fab modules above? Well, this is a typical example. PPC devtools ship with modules like this out of the box) so things like mouse, keyboard and external peripherals wouldn’t work.
And like always, which is the curse of Amiga, “something came up”, and the whole Raspberry PI / ARM initiative was left for dead. The details around this is sketchy, but the lead developer had a personal issue that forced him to set a new direction in life. And for some reason the other Aros developers have just continued with x86, even though a polished ARM version could have made them some money, and helped finance future development. It’s the same story, again and again.
But then something amazing happened! Out of the blue came Pascal Papara with a new take on Aros, namely AEROS. This is a distro after my own heart. Instead of wasting years trying to implement everything from scratch, Pascal took the Linux kernel and ran with it. So Aeros boots by virtue of the Linux Kernel, but jumps straight into Aros once the drivers has loaded. And the result? It is the fastest desktop you will ever experience on ARM. Seriously, it runs so fast and smooth on the Raspberry PI that you could easily mistake it for a $450 Intel i3.
Sadly Pascal has been more or less alone about this development. And truth be told he has molded it to suit his own needs rather than the consumer. Since his work includes a game machine and some Linux services, the whole Linux system is exposed to the Aros desktop. This is a huge mistake.
Using the Linux kernel to capitalize on the thousands of man hours invested in that, not to mention the linux driver database which is massive, is a great idea. It’s also the first thing that came into my mind when contemplating the issue.
But when running Aros on top of this, the Linux aspect of the system should be abstracted away. Much like what Apple did with Unix. You should hardly notice that Linux is there unless you open a shell and start to investigate. The Amiga filesystem should be the only filesystem you see when accessing things from the desktop, and a nice preferences option for showing / hiding mounted Linux drives.
My plans for an ARM based Amiga inspired OS
Building an OS is not a task for the faint of heart. Yes there is a lot of embedded / pre-fab based systems to pick from out there, but you also have to be sensible. You are not going to code a better kernel than Linus Torvalds, so instead of wasting years trying to catch up with something you cannot possibly catch up with – just grab the kernel and make it work for us.
The Linux kernel solves things such as process contexts, “userland” vs “kernel space” (giving the kernel the power to kill a task and reclaim resources), multitasking / threading, thread priorities, critical sections, mutexes and global event objects; it gives us IPC (inter process communication), disk IO, established and rock solid sound and graphics frameworks; and last but perhaps most important: free access to the millions of drivers in the Linux repository.
You would have to be certified insane to ignore the Linux Kernel, thinking you will somehow be the guy (or group) that can teach Linus Torvalds a lesson. This is a man who has been writing kernel’s for 20+ years, and he does nothing else. He is surrounded by a proverbial army of developers that code, test, refactor and strive to deliver optimal performance, safety and quality assurance. So sorry if I push your buttons here, but you would be a moron to take him on. Instead, absorb the kernel and gain access to the benefits it has given Linux (technically the kernel is “Linux”, the rest is GNU – but you get what I mean).
With the Linux kernel as a foundation, as much as 50% of the work involved in writing our OS is finished already. You don’t have to invent a driver API. You dont have to invent a new executable format (or write your own ELF parser if you stick with the Linux executable). You can use established compilers like GCC / Clang and Freepascal. And you can even cherry pick some low-level packages for your own native API (like SDL, OpenGL and things that would take years to finish).
But while we want to build our house on rock, we don’t want it to be yet another Linux distro. So with the kernel in place and a significant part of our work done for us, that is also where the similarities end.
The end product is Amiga OS, which means that we need compatibility with the original Amiga rom libraries (read: api). Had we started from scratch that would have been a tremendous effort, which is also why Aros is so important. Because Aros gives us a blueprint of how they have implemented these API’s.
But our main source of inspiration is not Aros, but Amithlon. What we want to do is naturally to pipe as much as we can from the Amiga API’s back to the Linux kernel. Things like device detection, memory allocation, file IO, pipes, networking — our library files will be more thin wrappers that expose Amiga compatible calls; methods that calls the Linux Kernel to do the job. So our Amiga library files will be proxy objects whenever possible.
The hard work is when we get to the window manager, or Intuition. Here we can’t cheat by pushing things back to Linux. We don’t want to install X either (although we can render our system into the X framebuffer if we like), so we have to code a window manager. This is not as simple as it sounds, because our system must operate with multiple cores, be multi threaded by design and tap into the grand scheme of things. Things like messages (which is used by applications to respond to input) must be established, and all the event codes from the original Amiga OS must be replicated.
So this work wont be easy, but with the Linux kernel as a foundation – the hardest task of all is taken care of. The magic of a kernel is that of process management and task switching. This is about as hard-core as you can get. Without that you can almost forget the rest. But since we base our system on the Linux kernel, we can focus 100% on the real task – namely to deliver a modern Amiga experience, one that is platform independent (read: conforms to standard Linux and can thus be recompiled and run anywhere Linux can run), preserves as much of the initial formula as possible – and can be successfully maintained far into the future.
By pushing as much of our work as possible into user-space (the process space where ordinary programs run, the kernel runs outside this space and is thus unaffected when a program crashes) and adhering to the Linux kernel beneath the bonnet, we have created a system that can be re-compiled anywhere Linux is. And it can be done so without any change to our codebase. Linux takes care of things like drivers, OpenGL, Sound — and presents to us a clean API that is identical on every platform. It doesn’t matter if it’s ARM, PPC, 68k, x86 or MIPS. As long as we follow the standards we are home free.
Last words
I hope all of this clears up the confusion that has surrounded the subject this week. Again, the misunderstanding that led to some unfortunate posts has been resolved. So there is no negativity, no drama and we are all on the same page.
Just remember that I have set some restrictions for my involvement here. I sincerely hope Hyperion and the Aros development group can focus on ARM, because the community needs this. While the Raspberry PI might seem too small a form-factor to run Aros, projects like Aeros have proven just how effective the Amiga formula is. I’m sure Hyperion could find a powerful ARM SOC in the price range of $120 and sell a complete package with profit for around $200.
What the Amiga community needs now, is not expensive hardware. The userbase has to be expanded horizontally across platforms. Amiga OS / Aros has much to offer the embedded market which today is dominated by overly complex Linux libraries. The Amiga can grow laterally as a more user-friendly alternative, much like Android did for the mobile market. Once the platform is growing and established – then custom hardware could be introduced. But right now that is not what we need.
I also hope that the Aros team drops whatever they are working on, fork Pascal Paparas codebase, and spend a few weeks polishing the system. Abstract away the Linux foundation like Apple have done, get those sexy 32 bit OS4 icons (Note: The icons used by Amiga OS 4 is available for free download from the designer’s website) and a nice theme that looks similar to OS 4 (but not too similar). Get Lazarus (the freepascal IDE) going and ship the system with a ready to use Pascal, C/C++ and Basic development environments. Bring back the fun in computing! The code is already there, use it!
Just take something simple, like a compatible browser. It’s actually not that simple, both for reasons of complexity and how memory is handled by PPC. With a Linux foundation things like Chromium Embedded could be inked into the Amiga side of things and we would have a native, fast, established and up-to-date browser.
At the same time, since we have API level compatability, people can recompile their Aros and Morphos applications and they would run more or less unchanged.
I really hope that my little protest here, if nothing else, helps people realize that there are viable options readily at hand. Commodore is not coming back, and the only future this platform has – is the one we make. So people have to ask themselves how much they want a future.
If the OS gains momentum then there will be grounds for investors to look at custom hardware. They can then choose off the shelves parts that are inexpensive to cover the normal functionality you expect in a modern computer – which more resources can go into custom hardware that sets the system apart. But we cant start there. It has to be built up brick by brich, standing on the shoulders of giants.
OK, rant over 🙂
Why buy a Vampire accelerator?
With the Amiga about to re-enter the consumer market, a lot of us “old timers” are busy knocking dust of our old machines. And I love my old machines even though they are technically useless by modern standards. But these machines have a lot of inspiration in them, especially if you write code. And yes there is a fair bit of nostalgia involved in this, there is no point in lying about any of this.
I mean, your mobile phone is probably 100 times faster than a vintage Amiga. But like you will discover with the new machines that are about to hit the market, there is more to this computer than you think. But vintage Amiga? Sadly they lack the power to anything useful [in the “modern” sense].
Enter the vampire
The Vampire is a product that started shipping about a year ago. It’s a FPGA based accelerator, and it’s quite frankly turning the retro scene on its head! Technically it’s a board that you just latch onto the CPU socket of your classical Amiga; it then takes over the whole machine and replace the CPU and chipset with its versions of these. Versions that are naturally a hell of a lot faster!
The result is that the good old Amiga is suddenly beefy enough to play Doom, Quake, MP3 files and MPG video (click here to read the datasheet). In short: this little board gives your old Amiga machine a jolt of new life.
Emulation vs. FPGA
Im not going to get into the argument about FPGA not being “real”, because that’s not what FPGA is about. Nor am I negative to classical hardware – because I own a ton of old Amiga gear myself. But I will get in your face when it comes to buying a Vampire.
Before we continue I just want to mention that there are two models of the vampire. There is the add-on board I have just mentioned which is again divided into different models for various Amiga versions (A600, A500 so far). The second model is a completely stand-alone vampire motherboard that wont even need a classic Amiga to work. It will be, for all means and purposes, a stand alone SBC (single board computer) that you just hook up power, video, storage and mouse – and off you go!
This latter version, the stand-alone, is a project I firmly believe in. The old boards have been out of production since 1993 and are getting harder to come by. And just like people they will eventually break down and stop working. There is also price to consider because getting your 20-year-old A500 fixed is not easy. First of all you need a specialist that knows how to fix these old things, and he will also need parts to work with. Since parts are no longer in production and homebrew can only go so far, well – a brand new motherboard that is compatible in every way sounds like a good idea.
There is also the fact that FPGA can reach absurd speeds. It has been mentioned that if the Vampire used a more expensive FPGA modules, 68k based Amiga’s could compete with modern processors (Source: https://www.generationamiga.com/2017/08/06/arria-10-based-vampire-could-reach-600mhz/). Can you imagine a 68k Amiga running side by side with the latest Intel processors? Sounds like a lot of fun if you ask me !
But then there is emulation. Proper emulation, which for Amiga users can only mean one thing: UAE in all its magnificent diversity and incarnations.
Nothing beats firing up a real Amiga, but you know what? It has been greatly exaggerated. I recently bought a sexy A1000 which is the first model that was ever made. This is the original Amiga, made way back before Commodore started to mess around with it. It cost me a small fortune to get – but hey, it was my first ever Amiga so I wanted to own one again.
But does it feel better than my Raspberry PI 3b powered A500? Nope. In fact I have only fired up the A1000 twice since I bought it, because having to wait for disks to load is just tedious (not to mention that you can’t get new, working floppy disks anymore). Seriously. I Love the machine to bits but it’s just damn tedious to work on in 2017. It belongs to the 80s and no-one can ever take away its glory or it’s role in computer history. That achievement stands forever.
High Quality Emulation
If you have followed my blog and Amiga escapades, you know that my PI 3b based Amiga, overclocked to the hilt, yields roughly 3.2 times the speed of an Amiga 4000/040. This was at one point the flagship Commodore computer. The Amiga 4000’s were used in movie production, music production, 3d rendering and heavy-duty computing all over the world. And the 35€ Raspberry PI gives you 3.2 times the power via the UAE4Arm emulator. I don’t care what the vampire does, the PI will give it the beating of its life.
Then suddenly, out of the blue, Asus comes along with the Tinkerboard. A board that I hated when it first came out (read part-1 here, part-2 here) due to its shabby drivers. The boards have been collecting dust on my office shelf for six months or so – and it was blind luck that i downloaded and tested a new disk image. If you missed that part you can read the full article here.
And I’m glad I did because man – the Tinkerboard makes the Raspberry PI 3b look like a toy! Asus has also adjusted the price lately. It was initially priced at 75€, but in Norway right now it retails for about 620 NKR – or 62€. So yes, it’s about twice the price of the PI – but it also gives you twice the memory, twice the graphics performance, twice the IO performance and a CPU that is a pleasure to work with.
The Raspberry PI 3b can’t be overclocked to the extent the model 1 and 2 could. You can over-volt it and tweak the GPU and memory and make it run faster. But people don’t call that “overclock” in the true sense of the word, because that means the CPU is set to run at speeds beyond the manufacturing specifications. So with the PI 3b there is relatively little you can do to make it run faster. You can speed it up a little bit, but that’s it. The Tinkerboard can be overclocked to the hilt.
Out of the box it runs at 1.5 Ghz, but if you add a heatsink, fan (important) and a 3A PSU – you can overclock it to 2.6 Ghz. And like the PI you can also tweak memory and gpu. So the Tinkerboard will happily run 3 times faster than the PI. If you add a USB3 harddisk you will also beef up IO speeds by 100 megabyte a second – which makes a huge difference. Linux does memory paging and it slows down everything if you just use the SD card.In short: if you fork out 70€ you get a SBC that runs rings around both the vampire and the Raspberry PI 3b. If we take height for some Linux services and drivers that have to run in the background, 3.2 x 3 = 9.6. Lets round that off to 9 since there will be performance hits by the background services. But still — 70€ for an Amiga that runs 9 times faster than A4000 @ MC68040 cpu ? That should blow your mind!
I’m sorry but there has to be something wrong with you if that doesn’t get your juices flowing. I rarely game on my classic Amiga setup. I’m a coder – but with this kind of firepower you can run some of the biggest and best Amiga titles ever made – and the Tinkerboard wont even break a sweat!
You can’t afford to be a fundamentalist
There are some real nutbags in the Amiga community. I think we all agree that having the real deal is a great experience, but the prices we see these days are borderline insane. I had to fork out around 500€ to get my A1000 shipped from Belgium to Norway. Had tax been added on the original price, I would have looked at something in the 700€ range. Still – 500€ for a 20-year-old computer that can hardly run Workbench 1.2? Unless you add the word “collector” here you are in fact barking mad!
If you are looking to get an Amiga for “old times sakes”, or perhaps you have an A500 and wonder if you should fork out for the Vampire? Will it be worth the 300€ pricetag? Unless you use your Amiga on a daily basis I can’t imagine what you need a vampire for. The stand-alone motherboard I can understand, that is a great idea – but the accelerator? 300€?
I mean you can pay 70€ and get the fastest Amiga that ever existed. Not a bit faster, not something on second place – no – THE FASTEST Amiga that has ever existed. If you think playing MP3 and MPG media files is cool with the vampire, then you are in for a treat here because the same software will work. You can safely download the latest patches and updates to various media players on the classic Amiga, and they will run just fine on UAE4Arm. But this time they will run a hell of a lot faster than the Vampire.
You really can’t be a fundamentalist in 2017 when it comes to vintage computers. And why would you want to? With so much cool stuff happening in the scene, why would you want to limit your Amiga experience to a single model? Aros is doing awesome stuff these days, you have the x5000 out and the A1222 just around the corner. Morphos is stable and good on the G5 PPC — there has never been a time when there were so many options for Amiga enthusiasts! Not even during the golden days between 1989-1994 were there so many exciting developments.
I love the classic Amiga machines. I think the Vampire stand-alone model is fantastic and if they ramp up the fpga to a faster model, they have in fact re-created a viable computer platform. A 68080 fpga based CPU that can go head to head with x86? That is quite an achievement – and I support that whole heartedly.
But having to fork out this amount of cash just to enjoy a modern Amiga experience is a bit silly. You can actually right now go out and buy a $35 Raspberry PI and enjoy far better results than the Vampire is able to deliver. How that can be negative? I have no idea, nor will I ever understand that kind of thinking. How do any of these people expect the Amiga community to grow and get new, young members if the average price of a 20-year-old machine costs 500€? Which incidentally is 50€ more than a brand new A1222 PPC machine capable of running OS 4.
And with the Tinkerboard you can get 9 times the speed of an A4000? How can that not give you goosebumps!
People talk about Java and Virtual-Machines like its black magic. Well UAE gives you a virtual CPU and chipset that makes mince-meat of both Java and C#. It also comes with one of the largest software libraries in the world. I find it inconceivable that no-one sees the potential in that technology beyond game playing – but when you become violent or nasty over hardware, then I guess that explains quite a bit.
I say, use whatever you can to enjoy your Amiga. And if your perfect Amiga is a PI or a Tinkerboard (or ODroid) – who cares!
I for one will not put more money into legacy hardware. I’m happy that I have the A1000, but that’s where it stops for me. I am looking forward to the latest Amiga x5000 PPC and cant wait to get coding on that – but unless the Appollo crew upgrades to a faster FPGA I see little reason to buy anything. I would gladly pay 500 – 1000 € for something that can kick modern computers in the behind. And I imagine a lot of 68k users would be willing to do that as well. But right now PPC is a much better option since it gives you both 68k and the new OS 4 platform in one price. And for affordable Amiga computing, emulation is now of such quality that you wont really notice the difference.
And I love coding 68k assembler on my Amibian emulator setup. There is nothing quite like it 🙂
Amibian.js on bitbucket
The Smart Pascal driven desktop known as Amibian.js is available on bitbucket. It was hosted in a normal github repository earlier – so make sure you clone out from this one.
About Amibian.js
Amibian is a desktop environment written in Smart Pascal. It compiles to JavaScript and can be used through any modern HTML5 compliant browser. The project consists of both a client and server, both written in smart pascal. The server is executed by node.js (note: please install PM2 to have better control over scaling and task management: http://pm2.keymetrics.io/).
Amibian.js is best suited for embedded projects, such as kiosk systems. It has been used in tutoring software for schools, custom routers and a wide range of different targets. It can easily be molded into a rich environment for SAD (single application devices) based software – but also made to act more as a real operating system:
- Class driven filesystem, easy to target external services
- Ram device-type
- Browser cache device-type
- ZIPfile device-type
- Node.js device-type
- Cross domain application hosting
- Traditional IPC protocol between hosted application and desktop
- Shared resources
- css styling
- glyphs and images
- Event driven visual controls
- Windowing manager makes it easy to implement custom applications
- Support for fullscreen API
Amibian ships with UAE.js (based on the SAE.js codebase) making it possible to run Amiga software directly on the desktop surface.
The bitbucket repository is located here: https://bitbucket.org/hexmonks/client
Amibian.js and the Narcissus hack
Wow, I must admit that I never really thought Amibian.js would become even remotely as popular as it has – yet people respond with incredible enthusiasm to our endeavour. I was just told that an article at Commodore USA mentioned us – that an exposure to 37000 readers. Add that to the roughly 40.000 people that subscribe to my feeds around the world and I must say: I hope I code something worthy of your time!
But there is a lot of stuff on the list before it’s even remotely finished. This is due to the fact that im not just juggling one codebase here – im juggling 5 separate yet interconnected codebases at the same time (!). First there is the Smart Mobile Studio RTL (run time library) which represents the foundation. This gives me object-oriented, fully inheritance driven visual controls. This have roughly 5 years of work behind it.
On top of that you have the actual visual controls, like buttons, scrollbars, lists, css3 effect engines, tweening, database storage and a ton of low-level stuff. The browser have no idea what a window is for example, let alone how it should look or respond to users. So every little piece has to be coded by someone. And well, that’s what I do.
Next you have the workbench and operating-system itself. What you know as Amibian.js, Smart Workbench or Quartex media desktop – take your pick, but it’s already a substantial codebase spanning some 40 units with thousands of lines of code. It is divided into two parts: the web front-end that you have all seen; and the node.js backend that is not yet made public.
And on top of that you have the external stuff. Quake III didn’t spontaneously self-assemble inside the desktop, someone had to do some coding and make the two interface. Same with all the other features you have.
The worst so far (as in damn hard to get right) is the Ace text editor. Ace by itself is super easy to work with – but you may have noticed that we have removed it’s scrollbars and replaced these with Amiga scrollbars instead? That is a formidable challenge it its own right.
Whats on the menu this weekend?
I noticed that on Linux that text-selection was utterly messed up, so when you moved a window around – it would suddenly start selecting the title text of other elements around the desktop. This is actually a bug in the browser – not my code; but I still have to code around it. Which I have now done.
I also solved selection for the console window (or any “text” container. A window is made up of many parts and the content region can be inherited from and replaced), so that should now work fine regardless of browser and platforms. Ace theming also works, and the vertical scrollbar is responding as expected. Still need a few tweaks to move right, but that is easy stuff. The hard part is behind us thankfully.
Right now I’m working on ScummVM so that should be in place later today 🙂
Thats cool, but what motivates you?
Retro gaming is important, and we have to make it as easy for people to enjoy their retro gear without patent trolls ruining the fun. Im just so tired of how ruined the Amiga scene is by these (3 companies in particular).. thieves is the only word I can find that fits.
So fine! I will make my own. Come hell or high water. Free as a bird and untouchable.
So I have made some tools that will make it ridiculously easy for you to share, download and play your games online. Whenever you want, hosted where-ever you need and there is not a god damn thing people can do about it. When you realize how simple the hack is it will make you laugh. I came up with this ages ago and dubbed it Narcissus.
To understand the Narcissus hack, consider the following:
PNG is a lossless compression format, meaning that it doesn’t lose any information when compressed. It’s not like JPEG which scrambles the original and saves a faximile that tricks the eye. Nope, if you compress a PNG image you get the exact same out when you decode it (read: show it).
But who said we have to store pixels? Pixels are just bytes after all. In fact, why can’t we take a whole game disk or rom and store that inside a picture? Sure you can!
It’s amusing, I came up with this hack years ago. It has been a part of Smart Mobile Studio since the beginning.
You have to remember that retro games are super small compared to modern games. The average ADF file is what? 880kb or something like that? Well hold on to your hat buddy, because PNG can hold 64 megabyte of data! You can encode a decent Amiga hard disk image in 64 megabyte.
Can you guess what the picture on the right contains? This picture is actually ALL the Amiga rom files packed into a single image. Dont worry, I converted it to JPEG to mess up the data before uploading. But yes, you can now host not just the games as normal picture files, but also roms and whatever you like.
And the beauty of it – who the hell is going to find them? You can host them on Github, Google drive, Dropbox or right your blog — if you don’t have the encryption key the file is useless.
Snap, crackle and pop!
RSS Filesystem
You know RSS feeds right? If you sign up for a blog you automatically get a RSS feed. It’s basically just a list of your recent posts – perhaps with an extract from each article, a thumbnail picture and links to each post. RSS have been around for a decade or more. It’s a great way to keep track of news.
The second hack is that using the data-to-image-encoder you can store a whole read-only filesystem as a normal RSS feed. Always think outside the box!
Let’s say you have a game collection for your Amiga right? Lets say 200 games. Wouldnt it be nice to have all those games online? Just readily available regardless of where you may be? Without “you know who” sending you a nasty email?
Well, just encode your game as described above, include the data-picture in your WordPress post, and do that for each of your games. Since you can encrypt these images they will be worthless to others. But for you its a neat way of hosting all your games online for free (like WordPress or Blogger) and play them via Amibian or the patched UAE4Arm (ops, did I share that, sorry dirk *grin*) and you’re home free.
You know what’s really cool? For this part Amibian doesnt even need a server. So you can just save the Amibian.js html page on your phone and that’s all you need.
You must be logged in to post a comment.