Archive

Archive for the ‘JavaScript’ Category

Quartex Pascal: Nearing completion

October 5, 2021 Leave a comment

When developers talk about web development they usually mean creating web pages with the tools common for the web sphere. Web designer software is abundantly available online, from single-click page wizards to more ad-hoc, old school HTML / JavaScript editors. If there is something the world don’t need more of, It’s one-click website solutions.

The Quartex Cloud cluster server, running services written in Quartex Pascal

One challenge that haven’t been addressed until recent times in the web sphere, is that of programming language. JavaScript is a fun language, but it was never really designed for large-scale application development. As websites become more and more elaborate, the need for traditional programming languages and features started to surface. In many ways the past 15 years of browser evolution, has been all about JavaScript catching up with the needs of developers.

But are we really limited to JavaScript?

When it comes to language and web technology, it was C/C++ that became the second language of the internet via the introduction of Asm.js and eventually Webassembly. It took a long time for other languages to adopt the Webassembly binary format as a target. WebAssembly is a bytecode binary format consisting of low-level instructions, much like assembly for x86 processors. These instructions are converted into real machine code by the browser (via a process called JIT compilation), and as a result the performance of Webassembly is close to native code. Having said that, Webassembly comes with its own set of restrictions and challenges, especially when it comes to manipulating the DOM (the document object model, the elements that makes up a HTML document).

The Quartex Way

Back in 2010 I had a novel idea with regards to languages: what if we translate Delphi code on source level, and emit JavaScript instead? At the time there was no such thing as webassembly, and the closest thing to a binary format was Macromedia Flash. Without rehashing the story, I teamed up with Eric Grange from Creative IT in France, the maintainer of Delphi Web Script, and the end result was a compiler that would parse Object Pascal code, construct an AST (abstract symbol tree) which is a model that represents the entire program, and further convert that into optimized JavaScript.

In order for such a system to work properly, a whole new RTL (runtime-library) had to be created. All the functions, procedures and classes that Delphi provides would not magically compile to JavaScript. So someone had to sit down and implement classes and features that made sense for the browser, from TComponent all the way up to TCustomControl – but in a way that is compatible with HTML.

Quartex Pascal comes with a rich RTL that makes class based, component oriented development possible for the browser

It is out of this work that the Quartex Framework came into being, as a personal research and development framework dealing with web technology. Back in 2014 it was just a utility library, and it remained as such until 2019 when it became a fully functional RTL in its own right. An RTL with a wingspan from low-level binary data, all the way up to visual components and database connectivity. In 2020 it expanded to Node.js, which is a JavaScript scripting-host used to write servers and services. The Quartex framework as now a full stack RTL that radically cuts down on development time needed when writing websites, mobile applications or server technology.

The Quartex IDE

Delphi has a wonderful IDE that has been polished and evolved over almost 2 decades. It is possible to introduce new compilers and third party technology into that IDE, but Delphi is limited to native technology. The only way to integrate QTX with Delphi, would be to mimic the VCL or FMX in its entirety, so that class-names match and the form design files could be read and used by the Quartex Compiler.

While such a project would probably be easier, it also meant a massive compromise in terms of features and performance. As a native development system Delphi does things in a very specific way, and if I forced JavaScript and HTML to abide by those rules – we would lose the dynamic and flamboyant aspects of HTML5 and JavaScript. The performance would also be poor since the VCL (and consequently LCL) was never written for the browser or Node.js. A test I did on performance, comparing QTX compiled code with TMS compiled code demonstrates my point. TMS populates a listbox with roughly 1000 items in 2 seconds. QTX populates the same listbox with 20.000 items in 1.8 seconds.

The only reasonable way forward was to implement a separate IDE, one that dealt with web technology exclusively. And what better language to write such a system in than Delphi itself? I was actually thinking that Embarcadero might want to rekindle their HTML5 Builder, and let me do my magic on it. Quartex Pascal is in many ways what HTML5 Builder should have been, and it’s just getting started.

The Quartex IDE: The welcome screen showing a live RSS feed from BeginEnd.net, as well as recent projects.

Writing an IDE is a massive undertaking. It covers technologies such as code suggestion, form and container designer, communication protocol design, license management – and much, much more. The IDE has been worked on every weekend for a year, and the results are solid.

What is important with an IDE like this, is that it represents a broad foundation for further development. It is written to be highly modular, with everything neatly isolated in classes. If a particular feature requires adjustment, then refactoring that particular module is a straightforward task. Large applications have a tendency to become a mesh of spaghetti that only the original developer understands, something I have worked very hard to avoid. The source-code is available for backers on Patreon.

Quartex IDE: Form designer and HTML5 property editor dialog

Server Side Programming

Node.js is a scripting host based on Google’s V8 JavaScript engine, which runs outside the browser. It is designed to run from the command-line (read: standard scripting host) and gives developers all the features you expect from a native program, like raw file access, multi threading (read: Node operates with multi processes), servers and sockets, third party libraries and much more.

Being able to write both client and server from the same development system, a so called “full stack” development environment, is a great boon and opens up for deployment on enterprise level.

Quartex IDE: Writing a HTTP/S server is no more difficult than using Indy under Delphi

But being able to communicate across services and servers means that the IDE had to provide the tools for async network programming. Working with async code is not hard, but it can be difficult if your codebase does not take height for it.

To help simplify communication between servers, services or clients (read: browser and server, or locally as inter process communication) I wrote the Ragnarok message framework. The IDE now has a visual protocol designer which makes it extremely easy to design messages and complex datatypes that is used when communicating. The protocol designer takes your design and generates ready-to-use classes and units.

Quartex IDE: The protocol designer greatly simplifies async client/server models

Object Pascal as a web language

You might think that object pascal with its rigid rules and pure logic is too stiff for web development. It turns out that this was exactly what the browser needed, as a solid anchor to the otherwise “anything goes” reality of JavaScript. Eric Grange made a lot of changes to the dialect which allows Quartex Pascal to interface more easily with JS, such as partial classes, external classes, static (in the C++ / C# meaning of the word), support for lambdas, inline variables, anonymous procedures, records and classes – and finally support for the async and await keywords when working with promises.

Quartex Pascal approach the DOM as a programmer would WinAPI, and the result is rock solid applications

Object Pascal brings a clarity to web development that JavaScript and TypeScript simply lacks. It also introduces normal inheritance (like C/C++ and Delphi has), with abstract and virtual members. When you combine this with partial classes, you have a dialect that is extremely productive, and that takes on node.js and Javascript on its own terms.

Come join the fun

The Quartex Pascal project is nearing completion. It is not finished just yet, but I am aiming for a release of version 1.0 before xmas. Quartex Pascal is based on Patreon backing, which means those that back the project and contribute financially enjoys weekly builds and working closer with the author on shaping the system. Premium backers also have access to the source-code, with rights to modify and use the system for whatever they like, providing the no-compete clause is respected.

If you find Quartex Pascal interesting – why not become a backer?

Quartex Pascal will be free for schools and educational institutions, as well as for students, non-profit organizations and open-source development. For commercial use a symbolic fee of $300 is needed. The system is licensed as shareware in order to avoid an avalanche of clones, which can quickly kill a project.

Quartex Pascal Build 13b ships

September 30, 2020 4 comments

While it can come across as disingenuous, I frickin love this project! As a developer yourself you know that feeling, when you manage to unify various aspects of your program, so that it all fits just perfectly. And the way I implemented file-handling and editors is within that sweet spot.

What is new?

It’s been a couple of weeks since I posted here last, so the list of changes will be quite dramatic. I think its best to focus on the highlights or this post would become very long!

Ironwood license management

Up until 2018 one of my products was a component package called HexLicense. This is a component package for Delphi that provides serial number validation, license handling and (most importantly) serial number minting. The HexLicense components were sold commercially until 2018 when I took them off the market and open-sourced (access was via Patreon. It is now moved to the Quartex Pascal project instead).

Ironwood is now an intrinsic part of the RTL and IDE

Im not going to go into how difficult it is to produce thousands of distinctly different serial numbers based on seed data, but it’s no walk in the park.

The final implementation I made for license minting and validation, was called Ironwood. It took the engine behind HexLicense and took it to a completely new level, incorporating both obfuscation and number modulation.

Generating license-number batches is literally one mouse-click to achieve

Needless to say, Ironwood is now a part of the Quartex Pascal RTL. To make it easier to work with the IDE has a nice utility for generating license-numbers, loading and saving keys, exporting license number batches – and much more.

There is also a ready-to-rock node.js application that can generate keys from the command-line (which is good to invoke from a server or service, so that it executes as a separate process).

HTML structure provider

The IDE has a very clean internal architecture, where the actual work is isolated in a set of easy to understand classes. One of these classes is called a TIDEAstProvider class. This is a class whose job it is to parse and otherwise work with whatever content an editor has, and deliver symbolic information that can be displayed in the file-structure treeview.

The IDE provides structured parsing and also Tag suggestions. More clever functionality will be added as we move into the final phases.

Obviously we have an object pascal provider, which will quickly compile and generate an AST very quickly in memory. This is used to power both the structure treeview and the code-suggestion.

Next, we have the exact same provider for JavaScript. So when you open a JavaScript file, the file will be processed to produce an AST, and the symbol information will be displayed exactly like your object pascal is. So behavior between these are identical.

We now also have a HTML provider, with a CSS provider on the way. The HTML provider is still in its infancy, but its flexible enough to represent a good foundation to work with. So I will no doubt return to this task later to smarten the provider logic up, and better handle un-valid HTML and CSS.

Code suggestion

Code suggestion is a pretty standard function these days. We have had support for this under Object Pascal for a while now in the IDE (with JavaScript on the way).

Note: the code suggestion-box is un-styled at this point. Custom painting will be added once the core functionality is done.

The IDE displays both a structural view of the unit, as well as in-depth code suggestion

Code suggestion for HTML is now in place too. It needs a bit of polish since the rules for HTML are wildly different from a programming language, but common behavior like TAG suggestion is there — with attributes, properties and events to follow.

So even if you are not an object pascal developer, the IDE should be nice to work with for traditional JavaScript / HTML code.

Form Recognition

While we cannot activate the form-designer just yet, since we need more AST functionality to extract things like class properties and attributes “live” to be able to do that properly — we are getting really close to that milestone.

The IDE however now recognize form files, so if your unit has an accompanying DFM file, the IDE is smart enough to open up a form-page. Form pages are different from ordinary pascal pages, since they also have the form designer control on a sub-tab. More or less identical to Delphi and Lazarus.

When you open a form-unit, the IDE is smart enough to recognize it as a form, opening the file-pair up in a layout capable page, just like Delphi

It is going to be so nice to get the form-designer activated. Especially the stack-based layout, which makes scalable, dynamic layout easy to create and work with.

The QTX RTL also supports orientation awareness as a part of the visual component system. One of the first things you will notice when exploring the code, is that ReSize() ships in an Orientation parameter, so you can adjust your layout accordingly.

Help and documentation inside the IDE

The IDE now has a PDF viewer with search functionality built-in. So when you click on Help and Documentation, a tab which shows the documentation PDF opens. This makes it easy to read, learn and find the information you need fast.

The IDE now has it’s own PDF renderer, so you can read the documentation directly

Well, that was a brief overview of what has changed since last time!

Next update is, as always, the weekends. We tend to land on sundays for new binaries, but do issue hotfixes in the evenings (weekdays) if something critical shows up.

Come join the fun!

Want to support the project? All financial backers that donates $100+ get their name in the product, access to the full IDE source-code on completion, and access to the Quartex Media Desktop system (which is a complete web desktop with a clustered back-end,  compiled to JavaScript and running on node.js. Portable, platform and chipset independent, and very powerful).

A smaller sum monthly is also welcome. The project would not exist without members backing it with $100, $200 etc every month. This both motivates and helps me allocate hours for continuous work.

When the IDE is finished you will also have access to the IDE source-code through special dispensation. Backers have rights since they have helped create the project.

Your help matters! It pays for components, hours and above all, tools and motivation. In return, you get full access to everything and a perpetual license. No backers will ever pay a cent for any future version of Quartex Pascal. Note: PM me after donating so I can get you added to the admin group! Click here to visit paypal: https://www.paypal.com/paypalme/quartexNOR

All donations are welcome, both large and small. But donations over $100, especially reoccurring, is what drives this project forward.

Remember to send me a message on Facebook so I can add you to the Admin group: https://www.facebook.com/quartexnor/

Quartex Pascal, Build 10b is out for backers

July 26, 2020 5 comments

I am deeply moved by some of the messages I have received about Quartex Pascal. Typically from people who either bought Smart Mobile studio or have followed my blog over the years. In short, developers that like to explore new ideas; people who also recognize some of the challenges large and complex run-time libraries like the VCL, FMX and LCL face in 2020.

Since I started with all this “compile to JavaScript” stuff, the world has changed. And while I’m not always right – I was right about this. JavaScript will and have evolved into a power-house. Largely thanks to Microsoft killing Basic. But writing large scale applications in JavaScript, that is extremely time consuming — which is where Quartex Pascal comes in.

support2

Quartex Pascal evolves every weekend. There are at least 2 builds each weekend for backers. Why not become a backer and see the product come to life? Get instant access to new builds, the docs, and learn why QTX code runs so much faster than the alternatives?

A very important distinction

Let me first start by iterating what I mentioned in my previous post, namely that I am no longer involved with The Smart Company. Nor am I involved with Smart Mobile Studio. I realize that it can be difficult for some to separate me from that product, since I blogged and created momentum for it for more than a decade. But alas, life change and sometimes you just have to let go.

The QTX Framework which today has become a fully operational RTL was written by me back between 2013-2014 (when I was not working for the company due to my spinal injury). And the first version of that framework was released under an open-source license.

When I returned to The Smart Company, it was decided that to save time – we would pull the QTX Framework into the Smart RTL. Since I owned the QTX Framework, and it was open source, it was perfectly fine to include the code. The code was bound by the open source licensing model, so we broke no rules including it. And I gave dispensation that it could be included (although the original license explicitly stated that the units should remain untouched and separate, and only inherited from).

desktop_02

Quartex Media Desktop, a complete desktop system (akin to X Windows for Linux) written completely in Object Pascal, including a clustered, service oriented back-end. All of it done in Quartex Pascal  — a huge project in its own right. Under Quartex Pascal,  this is now a project type, which means you can have your own cloud solution at the click of a button.

As I left the company for good before joining Embarcadero, The Smart Company and myself came to an agreement that the parts of QTX that still exists in the Smart Mobile Studio RTL, could remain. It would be petty and small to make a huge number out of it, and I left on my own terms. No point ruining all that hard work we did. So we signed an agreement that underlined that there would be overlaps here and there in our respective codebases, and that the QTX Framework and Quartex Media Desktop was my property.

Minor overlaps

As mentioned there will be a few minor overlaps, but nothing substantial. The class hierarchy and architecture of the QTX RTL is so different, that 90% of the code in the Smart RTL simply won’t work. And I made it that way on purpose so there would be no debates over who made what.

QTX represents how I felt the RTL should have been done. And its very different  from where Smart Mobile Studio ended up.

The overlaps are simple and few, but it can be helpful for Smart developers to know about if they plan on taking QTX for a test-drive:

  • TInteger, TString and TVariant. These were actually ported from Delphi (part of the Sith Library, a pun on Delphi’s Jedi Library).
  • TDataTypeConverter came in through the QTX Framework. It has been completely re-written from scratch. The QTX version is endian aware (works on both ARM, X86 and PPC). Classes that deal with binary data (like TStream, TBuffer etc) inherit from TDataTypeConverter. That way, you dont have to call a secondary instance just to perform conversion. This is easier and much more efficient.
  • Low-level codecs likewise came from the QTX Framework, but I had to completely re-write the architecture. The old model could only handle binary data, while the new codec classes also covers text based formats. Codecs can be daisy-chained so you can do encoding, compression and encryption by feeding data into the first, and catching the processed data from the last codec in the chain. Very handy, especially when dealing with binary messages and database drivers.
  • The in-memory dataset likewise came from the QTX Framework. This is probably the only unit that has remained largely unchanged. So that is a clear overlap between the Smart RTL and QTX.
  • TextCraft is an open source library that covers Delphi, Freepascal and DWScript. The latter was pulled in and used as the primary text-parser in Smart. This is also the default parser for QTX, and have been largely re-written so it could be re-published under the Shareware license.

Since the QTX RTL is very different from Smart, I haven’t really bothered to use all of the old code. Stuff like the CSS Effects units likewise came from the QTX Framework, but the architecture I made for Smart is not compatible with QTX, so it makes no sense using that code. I ported my Delphi tweening library to DWScript in 2019, which was a part of my Patreon project. So most of the effects in QTX use our own tweening library. This has some very powerful perks, like being able to animate a property on any object rather than just a HTML Element. And you can use it for Canvas drawing too, which is nice.

Progress. Where are we now?

So, where am I in this work right now? The RTL took more or less 1 year to write from  scratch. I only have the weekends  for this type of work,  and it would have been impossible without my backers. So I cannot thank each backer enough for the faith in this. The RTL and new IDE is actually just a stopping point on the road to a much bigger project, namely CloudForge, which is the full IDE running as an application on the Quartex Media Desktop. But first, let’s see what has been implemented!

AST unit view

unit_view

The Unit Overview panel. Easy access to ancestor classes as links (still early R&D). And the entire RTL on a second tab. This makes it very easy to learn the new RTL. There is also proper documentation, both as PDF and standard helpfile.

When the object-pascal code is compiled by DWScript, it goes through a vigorous process of syntax checking, parsing, tokenizing and symbolization (or objectification is perhaps a better word), where every inch of the code is transformed into objects that the compiler can work with and produce code from. These symbols are isolated in what is known as an AST, short for “Abstract Symbol Tree”. Basically a massive in-memory tree structure that contains your entire program reduced to symbols and expressions.

In order for us to have a live structural view of the current unit, I have created a simple background process that compiles the current unit, grabs the resulting AST, locates the unit symbol, and then displays the information in a meaningful way. This is the exact same  as most other IDE’s do, be it Visual Studio, Embarcadero Delphi, or Lazarus.

So we have that in place already. I also want to make it more elaborate, where  you can click yourself to glory by examining ancestors, interfaces, partial class groups – as well as an option to include inherited members (which should be visually different). Either way, the AST code is done. Just need to consolidate a few tidbits so that each Treeview node retains information about source-code location (so that when you double-click a symbol, the IDE navigates to where the symbol exists in the codebase).

JavaScript parsing and compilation

QTX doesn’t include just one compiler, but three. In order for the unit structure to also work for JavaScript files I have modified Besen, which is an ES5 compatible JavaScript engine written in Delphi. And I use this much like DWScript to parse and work with the AST.

unit_view2

Besen is a wonderful compiler. Where DWScript produces JavaScript from Object Pascal, Besen produces bytecodes from JavaScript (which are further JIT compiled). This opens up for some interesting options. I need to add support for ES6 though, modules and require are especially important for modern node.js programming (and yes, the QTX RTL supports these concepts)

HTML5 Rendering and CSS preview

Instead of using Chromium inside the IDE, which is pretty demanding, I have decided to go for HTMLComponents to deal with “normal” tasks. The “Welcome” tab-page for example — it would be complete overkill to use a full Chromium instance just for that, and TEdgeBrowser is likewise shooting sparrows with a Bazooka.

THTMLComponents have a blistering fast panel control that can render more or less any HTML5 document you throw at it (much better than the old TFrameViewer component). But obviously, it doesn’t have JS support. But we won’t be using JS when displaying static information – or indeed, editing HTML5 compliant content.

WYSIWYG Editor

The biggest benefit for HTMLComponents, is that it’s a fully operational HTML compliant editor. Which means you can do more or less all your manual design with that editor. In Quartex Pascal there is direct support for HTML files. Quartex works much like Visual Studio code, except it has visual designers. So you can create a HTML file and either type in the code manually, or switch to the HTMLComponents editor.

Which is what products like Help & Manual uses it for

helpmanual3

Image from HTMLComponents application gallery website

Support for HTML, CSS and JS files directly

While not new, this is  pretty awesome. Especially since we can do a bit of AST navigation here too to present similar information as we do for Object Pascal. The whole concept behind the QTX RTL, is that you have full control over everything. You can stick to a normal Delphi like form designer and absolute positioning, or you can opt for a more dynamic approach where you create content via code. This is perfect for modern websites that blend scrolling, effects and content (both dynamic and static content) for a better user experience.

You can even (spoiler alert), take a piece of HTML and convert it into visual controls at runtime. That is a very powerful function, because when doing large-scale, elaborate custom controls – you can just tell the RTL “hey, turn this piece of HTML into a visual control for me, and deliver it back when you are ready).

Proper Form Designer

Writing a proper form designer like Delphi has is no walk in the park. It has to deal not just with a selected control, but also child elements. It also has to be able to select multiple elements based on key-presses (shift + click adds another item to the selection),  or from the selection rectangle.

unit_view3

A property form layout control. Real-time rendering of controls is also possible, courtesy of HTMLComponents. But to be honest, it just gets in the way. Its much easier to work with this type of designer. It’s fast, responsive, accurate and will have animated features that makes it a joy to work with. 

Well, that’s not going to be a problem. I spent a considerable amount of time writing a proper form designer, one that takes both fixed and dynamic content into account. So the Quartex form designer handles both absolute and stacked layout modes (stacked means top-down, what in HTML is knock as blocking element  display, where each section stretch to the full width, and only have a defined height [that you can change]).

Node.js Service Protocol Designer

Writing large-scale servers, especially clustered ones, is very fiddly in vanilla JavaScript under node.js. It takes 3 seconds to create a server object, but as we all know, without proper error handling, a concurrent message format, modern security and a solid framework to handle it all — that “3 second” thing falls to the ground quickly.

This is where the Ragnarok message system comes in. Which is both a message framework, and a set of custom servers adapted for dealing with that type of data. It presently supports WebSocket, TCP and UDP. But expanding that to include REST is very easy.

support3

This is where the full might of the QTX Framework begins to dawn. As i wrote before we started on the Quartex Media Desktop (Which this IDE and RTL is a part of), in the future developers wont just drag & drop components on a form; they will drag & drop entire ecosystems ..

But the power of the system is not just in how it works, and how you can create your own protocols, and then have separate back-end services deal with one part of your infrastructure’s workload. It is because you can visually design the protocols using the Node Builder. This is being moved into the QTX IDE as I type. So should make it for Build 12 next weekend.

In short, you design your protocols, messages and types – a bit like RemObjects SDK if you have used that. And the IDE generates both server and client code for you. All you have to do is fill in the content that acts on the messages. Everything else is handled by the server components.

Suddenly, you can spend a week writing a large-scale, platform agnostic service stack — that would have taken JavaScript developers months to complete. And instead of having to manage a 200.000 lines codebase in JavaScript — you can enjoy a 4000 line, easily maintainable Object Pascal codebase.

Build 11

Im hoping to have build 11 out tomorrow (Sunday) for my backers. Im still experimenting a bit with the symbol information panel, since I want it to be informative not just for classes, but also for methods and properties. Making it easy to access ancestor implementations etc.

I also need to work a bit more on the JS parsing. Under ES5 its typical to use variables to hold objects  (which is close to how we think of a class), so composite and complex datatypes must be expanded. I  also need to get symbol position to work property, because since Besen is a proper bytecode compiler, it doesn’t keep as much information in it’s AST as DWScript does.

Widgets (which is what visual controls are called under QTX) should appear in build 12 or 13. The IDE supports zip-packages. The file-source system I made for the TVector library (published via Embarcadero’s website a few months back) allows us to mount not just folders as a file-source, but also zip files. So QTX component packages will be ordinary zip-files containing the .pas files, asset files and a metadata descriptor file that tells the IDE what to expect. Simple,  easy and very effective.

Support the project!

Want to support the project? All financial backers that donates $100+ get their name in the product, access to the full IDE source-code on completion, and access to the Quartex Media Desktop system (which is a complete web desktop with a clustered back-end,  compiled to JavaScript and running on node.js. Portable, platform and chipset independent, and very powerful).

support

Your help matters! It pays for components, hours and above all, tools and motivation. In return, you get full access to everything and a perpetual license. No backers will ever pay a cent for any future version of Quartex Pascal. Note: PM me after donating so I can get you added to the admin group! Click here to visit paypal: https://www.paypal.com/paypalme/quartexNOR

All donations are welcome, both large and small. But donations over $100, especially reoccurring, is what drives this project forward. It also gets you access to the Quartex Developer group on Facebook where new builds, features etc is happening. It is the best way to immediately get the latest build, read documentation as its written and see the product come to life!

 

Quartex Pascal, convergence is near

July 16, 2020 1 comment

67582488_10156396548830906_5204248427029856256_o

A Quartex Cluster of 5 x ODroid XU4. A $400 super computer running Quartex media Desktop. Enough to power a school.

I only have the weekends to work on Quartex Pascal, but I have spent the past 18 months tinkering away, making up for wasted time. So I’m just going to leave some pictures here for you to enjoy.

Note: I was asked on LinkedIn if this has anything to do with Smart Mobile Studio, and the answer is a resounding no. I have nothing to do with Smart any more. QTX Pascal is a completely separate project that is written from scratch by yours truly.

The QTX Framework was initially a library I created back in 2014, but it has later been completely overhauled and turned into a full RTL. It is not compatible with Smart Pascal and has a completely different architecture.

QTX Pascal is indirectly funded by the Amiga Retro Community (which might sound strange, but the technical level of that community is beyond anything I have encountered elsewhere) since QTX is central to the creation of the Quartex Media Desktop. It is a shame that Embarcadero decided to not back the project. The compiler and toolchain would have been a part of Delphi by now, and I wouldn’t have to write a separate IDE. But when they see what this system can deliver in terms of services, database work, mobile and embedded -they might regret it. The project only accepts donation funding, I am not interested in investors or partners. If you want a vision turned into reality, you gotta do it yourself. Everything else just gets in the way.

For developers by developers

Quartex Pascal is made for the community. It will be free for students and open-source projects. And a commercial license will never exceed $300. It is a shareware license and the financial aspects is purely to help fund further research and development of the desktop cloud platform. The final goal (CloudForge) is to compile the IDE itself to JavaScript, so people only need a browser to write enterprise level applications via Quartex Media Desktop. When that is finished, my work is done – and people have a clear path to the future.

qtx_run_07

Unlike other systems, QTX started with the non-visual stuff, so the system has a well implemented infrastructure for writing universal services and servers, using node.js as a deployment host. Services are also Docker friendly. Runs without change on Windows, Mac OS, Linux and a wealth of embedded systems and SBCs (single board computers)

qtx_run_08

A completely new RTL written from scratch generates close to native speed JS, highly compatible (even with legacy browsers) and rock solid

qtx_run_09

There are several display modes for QTX forms, from dynamic to absolute positioning. You can mix and match between HTML and QTX code, including a HTML5 compliant WYSIWYG editor and style manager. Makes content handling a lot easier

qtx_run_10

Write object pascal, JavaScript, HTML, LDEF (webassembly), node.js services – or mix and match between them all for maximum potential. Writing mobile applications is now ridiculously easy compared to “other tools” out there.

Oh and for the proverbial frosting — The full clustered Quartex Media desktop and services is a project type. Thats right. A complete cloud infrastructure suitable for teams, kiosks, embedded, schools, intranets – and even an replacement OS for ChromeOS. You don’t need to interface with Amazon, you get your own Amazon (optional naturally).

desktop_02

Filesystem over websocket, IPC between hosted apps and desktop, full back-end services that are clustered, and run on anything from a Raspberry PI 4 to low-cost ARM SBCs.

49938355_1169526123220996_502291013608407040_o

Web Assembly made easy. Both for Delphi and QTX

smartdesk

Let there be rock

Oh, and documentation. Loads and loads of documentation.

qtx_run_11

Proper documentation, both class overview and explanations that a human being has written is paramount for learning and getting up to speed quickly.

I don’t have vacation this year, which means I only have weekends to tinker away. But i have spent the past 18-ish months preparing and slowly finishing the pieces I needed. From vector containers to form design controls, to a completely re-written RTL from scratch — so yeah. This time I’m doing it my way.

C/C++ porting, QTX and general status

March 15, 2020 3 comments

C is a language that I used to play around with a lot back in the Amiga days. I think the last time I used a C compiler to write a library must have been in 1992 or something like that? I held on to my Amiga 1200 for as long as i could – but having fallen completely in love with Pascal, I eventually switched to x86 and went down the Turbo Pascal road.

Lately however, C++ developers have been asking for their own Developer group on Facebook. I run several groups on Facebook in the so-called “developer” family. So you have Delphi Developer, FPC Developer, Node.JS Developer and now – C++Builder developer. The groups more or less tend to themselves, and the node.js and FPC groups are presently being seeded (meaning, that the member count is being grown for a period).

The C++Builder group however, is having the same activity level as the Delphi group almost, thanks to some really good developers that post links, tips and help solve questions. I was also fortunate enough to have David Millington come on the Admin team. David is leading the C++Builder project, so his insight  and knowledge of both language and product is exemplary. Just like Jim McKeeth, he is a wonderful resource for the community and chime in with answers to tricky questions whenever he has time to spare.

Getting back in the saddle

Having working some 30 years with Pascal and Object Pascal, 25 of those years in Delphi, C/C++ is never far away. I have an article on the subject that i’ve written for the Idera Community website, so I wont dig too deep into that here — but needless to say, Rad Studio consists of two languages: Object Pascal and C/C++, so no matter how much you love either language, the other is never far away.

So I figured it was time for this old dog to learn some new tricks! I have always said that it’s wise to learn a language immediately below and above your comfort zone. So if Delphi is your favorite language, then C/C++ is below you (meaning: more low level and complex). Above you are languages like JavaScript and C#. Learning JavaScript makes strategic sense (or use DWScript to compile Pascal to JavaScript like I do).

When I started out, the immediate language below Object Pascal was never C, but assembler. So for the longest time I turned to assembler whenever I needed a speed boost; graphics manipulation and processing pixels is especially a field where assembly makes all the difference.

But since C++Builder is indeed an integral part of Rad Studio, and Object Pascal and C/C++ so intimately connected (they have evolved side by side), why not enjoy both assembly and C right?

So I decided to jump back into the saddle and see what I could make of it.

C/C++ is not as hard as you think

intf

I’m having a ball writing C/C++, and just like Delphi – you can start where you are.

While I’m not going to rehash the article I have already prepared for the Idera Community pages here, I do want to encourage people to give it a proper try. I have always said that if you know an archetypal language, you can easily pick up other languages, because the archetypal languages will benefit you for a lifetime. This has to do with archetypal languages operating according to how computers really work; as opposed to optimistic languages (a term from the DB work, optimistic locking), also called contextual languages, like C#, Java, JavaScript etc. are based on how human beings would like things to be.

So I now had a chance to put my money where my mouth is.

When I left C back in the early 90s, I never bothered with OOP. I mean, I used C purely for shared libraries anyways, while the actual programs were done in Pascal or a hybrid language called Blitz Basic. The latter compiled to razor sharp machine code, and you could use inline assembly – which I used a lot back then (very few programmers on those machines went without assembler, it was almost given that you could use 68k in some capacity).

Without ruining the article about to be published, I had a great time with C++Builder. It took a few hours to get my bearings, but since both the VCL and FMX frameworks are there – you can approach C/C++ just like you would Object Pascal. So it’s a matter of getting an overview really.

Needless to say, I’ll be porting  a fair share of my libraries to C/C++ when I have time (those that makes sense under that paradigme). It’s always good to push yourself and there are plenty of subtle differences that I found useful.

Quartex Media Desktop

When I last wrote about QTX we were nearing the completion of the FileSystem and Task Management service. The prototype had all its file-handling directly in the core service  (or server) which worked just fine — but it was linked to the Smart Pascal RTL. It has taken time to write a new RTL + a full multi-user, platform independent service stack and desktop (phew!) but we are seeing progress!

desktop

The QTX Baseline backend services is now largely done

The filesystem service is now largely done! There are a few synchronous calls I want to get rid of, but thankfully my framework has both async and sync variations of all file procedures – so that is now finished.

To make that clearer: first I have to wrap and implement the functionality for the RTL. Once they are in the RTL, I can use those functions to build the service functions. So yeah, it’s been extremely elaborate — but thankfully it’s also become a rich, well organized codebase (both the RTL and the Quartex Media Desktop codebases) – so I think we are ready to get cracking on the core!

The core is still operating with the older API. So our next step is to remove that from the core and instead delegate calls to the filesystem to our new service. So the core will simply be reduced to a post-office or traffic officer if you like. Messages come in from the desktops, and the core delegates the messages to whatever service is in charge of them.

But, this also means that both the core and the desktop must use the new and fancy messages. And this is where I did something very clever.

While I was writing the service, I also write a client class to test (obviously). And the way the core works — means that the same client that the core use to talk to the services — can be used by the desktop as well.

So our work in the desktop to get file-access and drives running again, is to wrap the client in our TQTXDevice ancestor class. The desktop NEVER accesses the API directly. All it knows about are these device drivers (or object instances). Which is  how we solve things like DropBox and Google Drive support. The desktop wont have the faintest clue that its using Dropbox, or copying files between a local disk and Google Drive for example — because it only communicates with these device classes.

Recursive stuff

One thing that sucked about node.js function for deleting a folder, is that it’s recursive parameter doesn’t work on Windows or OS X. So I had to implement a full recursive deletefolder routine manually. Not a big thing, but slightly more painful than expected under asynchronous execution. Thankfully, Object Pascal allows for inline defined procedures, so I didn’t have to isolate it in a separate class.

Here is some of the code, a tiny spec compared to the full shabam, but it gives you an idea of what life is like under async conditions:

unit service.file.core;

interface

{.$DEFINE DEBUG}

const
  CNT_PREFS_DEFAULTPORT     = 1883;
  CNT_PREFS_FILENAME        = 'QTXTaskManager.preferences.ini';
  CNT_PREFS_DBNAME          = 'taskdata.db';

  CNT_ZCONFIG_SERVICE_NAME  = 'TaskManager';

uses
  qtx.sysutils,
  qtx.json,
  qtx.db,
  qtx.logfile,
  qtx.orm,
  qtx.time,

  qtx.node.os,
  qtx.node.sqlite3,
  qtx.node.zconfig,
  qtx.node.cluster,

  qtx.node.core,
  qtx.node.filesystem,
  qtx.node.filewalker,
  qtx.fileapi.core,

  qtx.network.service,
  qtx.network.udp,

  qtx.inifile,
  qtx.node.inifile,

  NodeJS.child_process,

  ragnarok.types,
  ragnarok.Server,
  ragnarok.messages.base,
  ragnarok.messages.factory,
  ragnarok.messages.network,

  service.base,
  service.dispatcher,
  service.file.messages;

type

  TQTXTaskServiceFactory = class(TMessageFactory)
  protected
    procedure RegisterIntrinsic; override;
  end;

  TQTXFileWriteCB = procedure (TagValue: variant; Error: Exception);
  TQTXFileStateCB = procedure (TagValue: variant; Error: Exception);

  TQTXUnRegisterLocalDeviceCB = procedure (TagValue: variant; DiskName: string; Error: Exception);
  TQTXRegisterLocalDeviceCB = procedure (TagValue: variant; LocalPath: string; Error: Exception);
  TQTXFindDeviceCB = procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception);
  TQTXGetDisksCB = procedure (TagValue: variant; Devices: JDeviceList; Error: Exception);

  TQTXGetFileInfoCB = procedure (TagValue: variant; LocalName: string; Info: JStats; Error: Exception);
  TQTXGetTranslatePathCB = procedure (TagValue: variant; Original, Translated: string; Error: Exception);

  TQTXCheckDevicePathCB = procedure (TagValue: variant; PathName: string; Error: Exception);

  TQTXServerExecuteCB = procedure (TagValue: variant; Data: string; Error: Exception);

  TQTXTaskService = class(TRagnarokService)
  private
    FPrefs:     TQTXIniFile;
    FLog:       TQTXLogEmitter;
    FDatabase:  TSQLite3Database;

    FZConfig:   TQTXZConfigClient;
    FRegHandle: TQTXDispatchHandle;
    FRegCount:  integer;

    procedure   HandleGetDevices(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
    procedure   HandleGetDeviceByName(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
    procedure   HandleCreateLocalDevice(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
    procedure   HandleDestroyDevice(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
    procedure   HandleFileRead(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
    procedure   HandleFileReadPartial(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
    procedure   HandleGetFileInfo(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
    procedure   HandleFileDelete(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);

    procedure   HandleFileWrite(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
    procedure   HandleFileWritePartial(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
    procedure   HandleFileRename(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
    procedure   HandleGetDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);

    procedure   HandleMkDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
    procedure   HandleRmDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);

    procedure   ExecuteExternalJS(Params: array of string;
      TagValue: variant; const CB: TQTXServerExecuteCB);

    procedure   SendError(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage; Message: string);

  protected
    function    GetFactory: TMessageFactory; override;
    procedure   SetupPreferences(const CB: TRagnarokServiceCB);
    procedure   SetupLogfile(LogFileName: string;const CB: TRagnarokServiceCB);
    procedure   SetupDatabase(const CB: TRagnarokServiceCB);

    procedure   ValidateLocalDiskName(TagValue: variant; Username, DeviceName: string; CB: TQTXCheckDevicePathCB);
    procedure   RegisterLocalDevice(TagValue: variant; Username, DiskName: string; CB: TQTXRegisterLocalDeviceCB);
    procedure   UnRegisterLocalDevice(TagValue: variant; UserName, DiskName:string; CB: TQTXUnRegisterLocalDeviceCB);

    procedure   GetDevicesForUser(TagValue: variant; UserName: string; CB: TQTXGetDisksCB);
    procedure   FindDeviceByName(TagValue: variant; UserName, DiskName: string; CB: TQTXFindDeviceCB);
    procedure   FindDeviceByType(TagValue: variant; UserName: string; &Type: JDeviceType; CB: TQTXGetDisksCB);

    procedure   GetTranslatedPathFor(TagValue: variant; Username, FullPath: string; CB: TQTXGetTranslatePathCB);

    procedure   GetFileInfo(TagValue: variant; UserName: string; FullPath: string; CB: TQTXGetFileInfoCB);

    procedure   SetupTaskTable(const TagValue: variant; const CB: TRagnarokServiceCB);
    procedure   SetupOperationsTable(const TagValue: variant; const CB: TRagnarokServiceCB);
    procedure   SetupDeviceTable(const TagValue: variant; const CB: TRagnarokServiceCB);

    procedure   AfterServerStarted; override;
    procedure   BeforeServerStopped; override;
    procedure   Dispatch(Socket: TNJWebSocketSocket; Message: TQTXBaseMessage); override;

  public
    property    Preferences: TQTXIniFile read FPrefs;
    property    Database: TSQLite3Database read FDatabase;

    procedure   SetupService(const CB: TRagnarokServiceCB);

    constructor Create; override;
    destructor  Destroy; override;
  end;


implementation

//#############################################################################
// TQTXFileenticationFactory
//#############################################################################

procedure TQTXTaskServiceFactory.RegisterIntrinsic;
begin
  writeln("Registering task interface");
  &Register(TQTXFileGetDeviceListRequest);
  &Register(TQTXFileGetDeviceByNameRequest);
  &Register(TQTXFileCreateLocalDeviceRequest);
  &Register(TQTXFileDestroyDeviceRequest);
  &Register(TQTXFileReadPartialRequest);
  &Register(TQTXFileReadRequest);
  &Register(TQTXFileWritePartialRequest);
  &Register(TQTXFileWriteRequest);
  &Register(TQTXFileDeleteRequest);
  &Register(TQTXFileRenameRequest);
  &Register(TQTXFileInfoRequest);
  &Register(TQTXFileDirRequest);
  &Register(TQTXMkDirRequest);
  &Register(TQTXRmDirRequest);
  &Register(TQTXFileRenameRequest);
  &Register(TQTXFileDirRequest);
end;

//#############################################################################
// TQTXTaskService
//#############################################################################

constructor TQTXTaskService.Create;
begin
  inherited Create;
  FPrefs := TQTXIniFile.Create();
  FLog := TQTXLogEmitter.Create();
  FDatabase := TSQLite3Database.Create(nil);

  FZConfig := TQTXZConfigClient.Create();
  FZConfig.Port := 2292;

  self.OnUserSignedOff := procedure (Sender: TObject; Username: string)
  begin
    WriteToLogF("We got a service signal! User [%s] has signed off completely", [Username]);
  end;

  MessageDispatch.RegisterMessage(TQTXFileGetDeviceListRequest, @HandleGetDevices);
  MessageDispatch.RegisterMessage(TQTXFileGetDeviceByNameRequest, @HandleGetDeviceByName);
  MessageDispatch.RegisterMessage(TQTXFileCreateLocalDeviceRequest, @HandleCreateLocalDevice);
  MessageDispatch.RegisterMessage(TQTXFileDestroyDeviceRequest, @HandleDestroyDevice);

  MessageDispatch.RegisterMessage(TQTXFileReadRequest, @HandleFileRead);
  MessageDispatch.RegisterMessage(TQTXFileReadPartialRequest, @HandleFileReadPartial);

  MessageDispatch.RegisterMessage(TQTXFileWriteRequest, @HandleFileWrite);
  MessageDispatch.RegisterMessage(TQTXFileWritePartialRequest, @HandleFileWritePartial);

  MessageDispatch.RegisterMessage(TQTXFileInfoRequest, @HandleGetFileInfo);
  MessageDispatch.RegisterMessage(TQTXFileDeleteRequest, @HandleFileDelete);

  MessageDispatch.RegisterMessage(TQTXMkDirRequest, @HandleMkDir);
  MessageDispatch.RegisterMessage(TQTXRmDirRequest, @HandleRmDir);
  MessageDispatch.RegisterMessage(TQTXFileRenameRequest, @HandleFileRename);

  MessageDispatch.RegisterMessage(TQTXFileDirRequest, @HandleGetDir);
end;

destructor TQTXTaskService.Destroy;
begin
  // decouple logger from our instance
  self.logging := nil;

  // Release prefs + log
  FPrefs.free;
  FLog.free;
  FZConfig.free;
  inherited;
end;

procedure TQTXTaskService.SendError(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage; Message: string);
begin
  var reply := TQTXErrorMessage.Create(request.ticket);
  try
    reply.Code := CNT_MESSAGE_CODE_ERROR;
    reply.Routing.TagValue := Request.Routing.TagValue;
    reply.Response := Message;

    if Socket.ReadyState = rsOpen then
    begin
      try
        Socket.Send( reply.Serialize() );
      except
        on e: exception do
        WriteToLog(e.message);
      end;
    end else
      WriteToLog("Failed to dispatch error, socket is closed error");
  finally
    reply.free;
  end;
end;

procedure TQTXTaskService.ExecuteExternalJS(Params: array of string;
  TagValue: variant; const CB: TQTXServerExecuteCB);
begin
  var LTask: JChildProcess;

  var lOpts := TVariant.CreateObject();
  lOpts.shell := false;
  lOpts.detached := true;

  Params.insert(0, '--no-warnings');

  // Spawn a new process, this creates a new shell interface
  try
    LTask := child_process().spawn('node', Params, lOpts );
  except
    on e: exception do
    begin
      if assigned(CB) then
        CB(TagValue, e.message, e);
      exit;
    end;
  end;

  // Map general errors on process level
  LTask.on('error', procedure (error: variant)
  begin
    {$IFDEF DEBUG}
    writeln("error->" + error.toString());
    {$ENDIF}
    WriteToLog(error.toString());

    if assigned(CB) then
      CB(TagValue, "", nil);
  end);

  // map stdout so we capture the output
  LTask.stdout.on('data', procedure (data: variant)
  begin
    if assigned(CB) then
      CB(TagValue, data.toString(), nil);
  end);

  // map stderr so we can capture exception messages
  LTask.stderr.on('data', procedure (error:variant)
  begin
    {$IFDEF DEBUG}
    writeln("stdErr->" + error.toString());
    {$ENDIF}

    if assigned(CB) then
      CB(TagValue, "", nil);

    WriteToLog(error.toString());
  end);
end;

function TQTXTaskService.GetFactory: TMessageFactory;
begin
  result := TQTXTaskServiceFactory.Create();
end;

procedure TQTXTaskService.SetupService(const CB: TRagnarokServiceCB);
begin
  SetupPreferences( procedure (Error: Exception)
  begin
    // No logfile yet setup (!)
    if Error  nil then
    begin
      WriteToLog("Preferences setup: Failed!");
      WriteToLog(error.message);
      raise error;
    end else
    WriteToLog("Preferences setup: OK");

    // logfile-name is always relative to the executable
    var LLogFileName := TQTXNodeFileUtils.IncludeTrailingPathDelimiter( TQTXNodeFileUtils.GetCurrentDirectory );
    LLogFileName += FPrefs.ReadString('log', 'logfile', 'log.txt');

    // Port is defined in the ancestor, so we assigns it here
    Port := FPrefs.ReadInteger('networking', 'port', CNT_PREFS_DEFAULTPORT);

    SetupLogfile(LLogFileName, procedure (Error: Exception)
    begin
      if Error  nil then
      begin
        WriteToLog("Logfile setup: Failed!");
        WriteToLog(error.message);
        raise error;
      end else
      WriteToLog("Logfile setup: OK");

      SetupDatabase( procedure (Error: Exception)
      begin
        if Error  nil then
        begin
          WriteToLog("Database setup: Failed!");
          WriteToLog(error.message);
          if assigned(CB) then
            CB(Error)
          else
            raise Error;
        end else
        WriteToLog("Database setup: OK");

        if assigned(CB) then
          CB(nil);
      end);

    end);
  end);
end;

procedure TQTXTaskService.SetupPreferences(const CB: TRagnarokServiceCB);
begin
  var lBasePath := TQTXNodeFileUtils.GetCurrentDirectory;
  var LPrefsFile := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(LBasePath) + CNT_PREFS_FILENAME;

  if TQTXNodeFileUtils.FileExists(LPrefsFile) then
  begin
    FPrefs.LoadFromFile(nil, LPrefsFile, procedure (TagValue: variant; Error: Exception)
    begin
      if Error  nil then
      begin
        if assigned(CB) then
          CB(Error)
        else
          raise Error;
        exit;
      end;

      if assigned(CB) then
        CB(nil);
    end);

  end else
  begin
    var LError := Exception.Create('Could not locate preferences file: ' + LPrefsFile);
    WriteToLog(LError.message);
    if assigned(CB) then
      CB(LError)
    else
      raise LError;
  end;
end;

procedure TQTXTaskService.SetupLogfile(LogFileName: string;const CB: TRagnarokServiceCB);
begin
  // Attempt to open logfile
  // Note: Log-object error options is set to throw exceptions
  try
    FLog.Open(LogFileName);
  except
    on e: exception do
    begin
      if assigned(CB) then
      begin
        CB(e);
        exit;
      end else
      begin
        writeln(e.message);
        raise;
      end;
    end;
  end;

  // We inherit from TQTXLogObject, which means we can pipe
  // all errors etc directly using built-in functions. So here
  // we simply glue our instance to the log-file, and its all good
  self.Logging := FLog as IQTXLogClient;

  if assigned(CB) then
    CB(nil);
end;

procedure TQTXTaskService.FindDeviceByType(TagValue: variant; UserName: string; &Type: JDeviceType; CB: TQTXGetDisksCB);
begin
  UserName := username.trim().ToLower();
  if Username.length < 1 then
  begin
    WriteToLog("Failed to lookup disk, username was invalid error");
    var lError := EException.Create("Failed to lookup devices, invalid username");
    if assigned(CB) then
      CB(TagValue, nil, lError)
    else
      raise lError;
    exit;
  end;

  GetDevicesForUser(TagValue, Username,
  procedure (TagValue: variant; Devices: JDeviceList; Error: Exception)
  begin
    if Error  nil then
    begin
      if assigned(CB) then
        CB(TagValue, nil, Error)
      else
        raise Error;
      exit;
    end;

    var x := 0;
    while x < Devices.dlDrives.Count do
    begin
      if Devices.dlDrives[x].&Type  &Type then
      begin
        Devices.dlDrives.delete(x, 1);
        continue;
      end;
      inc(x);
    end;

    if assigned(CB) then
      CB(TagValue, Devices, nil);
  end);
end;

procedure TQTXTaskService.FindDeviceByName(TagValue: variant; Username, DiskName: string; CB: TQTXFindDeviceCB);
begin
  UserName := username.trim().ToLower();
  if Username.length < 1 then
  begin
    var lLogText := "Failed to lookup device, username was invalid error";
    WriteToLog(lLogText);
    var lError := EException.Create(lLogText);
    if assigned(CB) then
      CB(TagValue, nil, lError)
    else
      raise lError;
    exit;
  end;

  DiskName := DiskName.trim();
  if DiskName.length < 1 then
  begin
    var lLogText := "Failed to lookup device, disk-name was invalid error";
    WriteToLog(lLogText);
    var lError := EException.Create(lLogText);
    if assigned(CB) then
      CB(TagValue, nil, lError)
    else
      raise lError;
    exit;
  end;

  GetDevicesForUser(TagValue, Username,
  procedure (TagValue: variant; Devices: JDeviceList; Error: Exception)
  begin
    if Error  nil then
    begin
      if assigned(CB) then
        CB(TagValue, nil, Error)
      else
        raise Error;
      exit;
    end;

    DiskName := DiskName.trim().ToLower();
    var lDiskInfo: JDeviceInfo := nil;


    for var disk in Devices.dlDrives do
    begin
      if disk.Name.ToLower() = DiskName then
      begin
        lDiskInfo := disk;
        break;
      end;
    end;

    if assigned(CB) then
      CB(TagValue, lDiskInfo, nil);
  end);
end;

procedure TQTXTaskService.GetTranslatedPathFor(TagValue: variant; Username, FullPath: string; CB: TQTXGetTranslatePathCB);
begin
  var lParser := TQTXPathParser.Create();
  try
    var lInfo: TQTXPathData;
    if lparser.Parse(FullPath, lInfo) then
    begin
      // Locate the device for the path belonging to the user
      FindDeviceByName(TagValue, UserName, lInfo.MountPart,
      procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception)
      begin
        if Error  nil then
        begin
          if assigned(CB) then
            CB(TagValue, FullPath, '', Error)
          else
            raise Error;
          exit;
        end;

        if Device.&Type  dtLocal then
        begin
          var lError := EException.CreateFmt('Failed to translate path, device [%s] is not local error', [Device.Name]);
          if assigned(CB) then
            CB(TagValue, FullPath, '', Error)
          else
            raise Error;
          exit;
        end;

        // We want the path + filename, so we can append that to
        // the actual localized filesystem
        var lExtract := FullPath;
        delete(lExtract, 1, lInfo.MountPart.Length + 1);

        // Construct complete storage location
        var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory();
        lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices';
        lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim();
        lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + lExtract;

        // Return translated path
        if assigned(CB) then
          CB(TagValue, FullPath, lFullPath, nil);

      end);
    end else
    begin
      var lErr := EException.CreateFmt("Invalid path [%s] error", [FullPath]);
      if assigned(CB) then
        CB(TagValue, FullPath, '', lErr)
      else
        raise lErr;
    end;
  finally
    lParser.free;
  end;
end;

procedure TQTXTaskService.GetFileInfo(TagValue: variant; UserName, FullPath: string; CB: TQTXGetFileInfoCB);
begin
  var lParser := TQTXPathParser.Create();
  try
    var lInfo: TQTXPathData;
    if lparser.Parse(FullPath, lInfo) then
    begin
      // Locate the device for the path belonging to the user
      FindDeviceByName(TagValue, UserName, lInfo.MountPart,
      procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception)
      begin
        if Error  nil then
        begin
          if assigned(CB) then
            CB(TagValue, '', nil, Error)
          else
            raise Error;
          exit;
        end;

        case Device.&Type of
        dtLocal:
          begin
            // We want the path + filename, so we can append that to
            // the actual localized filesystem
            var lExtract := FullPath;
            delete(lExtract, 1, lInfo.MountPart.Length + 1);

            // Construct complete storage location
            var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory();
            lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices';
            lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim();
            lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + lExtract;

            // Call the underlying OS to get the file statistics
            NodeJsFsAPI().lStat(lFullPath,
            procedure (Error: JError; Stats: JStats)
            begin
              if Error  nil then
              begin
                var lError := EException.Create(Error.message);
                if assigned(CB) then
                  CB(TagValue, lFullPath, nil, lError)
                else
                  raise lError;
                exit;
              end;

              // And deliver
              if assigned(CB) then
                CB(TagValue, lFullPath, Stats, nil);
            end);
          end;
        dtDropbox, dtGoogle, dtMsDrive:
          begin
            var lError := EException.Create("Cloud bindings not activated error");
            if assigned(CB) then
              CB(TagValue, '', nil, lError)
          end;
        end;
      end);
    end else
    begin
      var lErr := EException.CreateFmt("Invalid path [%s] error", [FullPath]);
      if assigned(CB) then
        CB(TagValue, '', nil, lErr)
      else
        raise lErr;
    end;
  finally
    lParser.free;
  end;
end;

procedure TQTXTaskService.GetDevicesForUser(TagValue: variant; Username: string; CB: TQTXGetDisksCB);
begin
  UserName := username.trim().ToLower();
  if Username.length < 1 then
  begin
    WriteToLog("Failed to lookup devices, username was invalid error");
    var lError := EException.Create("Failed to lookup devices, invalid username");
    if assigned(CB) then
      CB(TagValue, nil, lError)
    else
      raise lError;
    exit;
  end;

  var lTransaction: TQTXReadTransaction;
  if not TSQLite3Database(DataBase).CreateReadTransaction(lTransaction) then
  begin
    var lErr := EException.Create("Failed to create read-transaction error");
    if assigned(cb) then
      CB(TagValue, nil, lErr)
    else
      raise lErr;
    exit;
  end;

  var lQuery := TSQLite3ReadTransaction(lTransaction);
  lQuery.SQL := "select * from devices where owner=?";
  lQuery.Parameters.AddValueOnly(Username);

  lQuery.Execute(
  procedure (Sender: TObject; Error: Exception)
  begin
    if Error  nil then
    begin
      if assigned(CB) then
        CB(TagValue, nil, Error)
      else
        raise Error;
      exit;
    end;

    var lDisks := new JDeviceList();
    lDisks.dlUser := UserName;

    for var x := 0 to lQuery.datarows.length-1 do
    begin
      var lInfo := new JDeviceInfo();
      lInfo.Name := lQuery.datarows[x]["name"];
      lInfo.&Type := JDeviceType( lQuery.datarows[x]["type"] );
      lInfo.owner := lQuery.datarows[x]["owner"];
      lInfo.location := lQuery.datarows[x]["location"];
      lInfo.APIKey := lQuery.datarows[x]["apikey"];
      lInfo.APISecret := lQuery.datarows[x]["apisecret"];
      lInfo.APIPassword := lQuery.datarows[x]["apipassword"];
      lInfo.APIUser := lQuery.datarows[x]["apiuser"];
      lDisks.dlDrives.add(lInfo);
    end;

    try
      if assigned(CB) then
        CB(TagValue, lDisks, nil);
    finally
      lQuery.free;
    end;
  end);
end;

procedure TQTXTaskService.ValidateLocalDiskName(TagValue: variant; Username, DeviceName: string; CB: TQTXCheckDevicePathCB);
begin
  var Filename := 'disk.' + username + '.' + DeviceName + '.' + ord(JDeviceType.dtLocal).ToString();

  var LBasePath := TQTXNodeFileUtils.GetCurrentDirectory();
  LBasePath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(LBasePath) + 'userdevices';

  // Make sure the device folder is there
  if not TQTXNodeFileUtils.DirectoryExists(LBasePath) then
  begin
    var lError := EException.CreateFmt("Directory not found: %s", [lBasePath]);
    if assigned(CB) then
      CB(TagValue, '', lError)
    else
      raise lError;
    exit;
  end;

  lBasePath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(LBasePath) + Filename;

  if TQTXNodeFileUtils.DirectoryExists(LBasePath) then
  begin
    var lError := EException.CreateFmt("Path already exist error [%s]", [lBasePath]);
    if assigned(CB) then
      CB(TagValue, '', lError)
    else
      raise lError;
    exit;
  end;

  // OK, folder is not created yet, so its good to go
  if assigned(CB) then
    CB(TagValue, Filename, nil);
end;

procedure TQTXTaskService.UnRegisterLocalDevice(TagValue: variant; UserName, DiskName: string; CB: TQTXUnRegisterLocalDeviceCB);
begin
  WriteToLogF("Removing local device [%s] for user [%s] ", [DiskName, Username]);

  // Check username string
  UserName := username.trim().ToLower();
  if Username.length < 1 then
  begin
    WriteToLog("Failed to unregister device, username was invalid error");
    var lError := EException.Create("Failed to register device, invalid username");
    if assigned(CB) then
      CB(TagValue, DiskName, lError)
    else
      raise lError;
    exit;
  end;

  // Check diskname string
  DiskName := DiskName.trim().ToLower();
  if DiskName.length < 1 then
  begin
    WriteToLog("Failed to unregister device, disk-name was invalid error");
    var lError := EException.Create("Failed to register device, invalid disk-name");
    if assigned(CB) then
      CB(TagValue, DiskName, lError)
    else
      raise lError;
    exit;
  end;

  FindDeviceByName(TagValue, Username, DiskName,
  procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception)
  begin
    // Did the search fail?
    if Error  nil then
    begin
      WriteToLog(Error.message);
      if assigned(CB) then
        CB(TagValue, DiskName, Error)
      else
        raise Error;
      exit;
    end;

    // Make sure the device is local
    if Device.&Type  dtLocal then
    begin
      var lError := EException.CreateFmt('Failed to translate path, device [%s] is not local error', [Device.Name]);
      if assigned(CB) then
        CB(TagValue, DiskName, Error)
      else
        raise Error;
      exit;
    end;

    // Delete record from database
    var lWriter: TQTXWriteTransaction;
    if FDatabase.CreateWriteTransaction(lWriter) then
    begin
      lWriter.SQL := "delete from profiles where user = ? and name = ?;";
      lWriter.Parameters.AddValueOnly(Username);
      lWriter.Parameters.AddValueOnly(DiskName);

      lWriter.Execute(
      procedure (Sender: TObject; Error: Exception)
      begin
        try

          if Error  nil then
          begin
            if assigned(CB) then
              CB(TagValue, DiskName, Error)
            else
              raise Error;
            exit;
          end;

          // Construct complete storage location
          var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory();
          lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices';
          lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim();

          // Now delete the disk-drive directory
          TQTXNodeFileUtils.DeleteDirectory(nil, lFullPath,
          procedure (TagValue: variant; Path: string; Error: Exception)
          begin
            if assigned(CB) then
              CB(TagValue, DiskName, Error)
          end);

        finally
          lWriter.free;
          lWriter := nil;
        end;
      end);
    end;
  end);
end;

procedure TQTXTaskService.RegisterLocalDevice(TagValue: variant; Username, DiskName: string; CB: TQTXRegisterLocalDeviceCB);
begin
  WriteToLogF("Adding local device [%s] for user [%s] ", [DiskName, Username]);

  UserName := username.trim().ToLower();
  if Username.length < 1 then
  begin
    WriteToLog("Failed to register device, username was invalid error");
    var lError := EException.Create("Failed to register device, invalid username");
    if assigned(CB) then
      CB(TagValue, '', lError)
    else
      raise lError;
    exit;
  end;

  DiskName := DiskName.trim().ToLower();
  if DiskName.length < 1 then
  begin
    WriteToLog("Failed to register device, disk-name was invalid error");
    var lError := EException.Create("Failed to register device, invalid disk-name");
    if assigned(CB) then
      CB(TagValue, '', lError)
    else
      raise lError;
    exit;
  end;

  FindDeviceByName(TagValue, Username, DiskName,
  procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception)
  begin
    // Did the search fail?
    if Error  nil then
    begin
      WriteToLog(Error.message);
      if assigned(CB) then
        CB(TagValue, '', Error)
      else
        raise Error;
      exit;
    end;

    // Does a device that match already exist?
    if Device  nil then
    begin
      var lError := EException.CreateFmt("Failed to create device [%s], device already exists", [DiskName]);
      if assigned(CB) then
        CB(TagValue, '', lError)
      else
        raise lError;
      exit;
    end;

    //  make sure the device-folder does not exist, so we can create it
    ValidateLocalDiskName(TagValue, Username, DiskName,
    procedure (TagValue: variant; PathName: string; Error: Exception)
    begin
      if Error  nil then
      begin
        if assigned(CB) then
          CB(TagValue, '', Error)
        else
          raise Error;
        exit;
      end;

      // ValidateLocalDiskName only returns the valid directory-name,
      // not a full path -- so we need to build up the full targetpath
      var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory();
      lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices';
      lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + PathName;

      TQTXNodeFileUtils.CreateDirectory(nil, lFullPath,
      procedure (TagValue: variant; Path: string; Error: exception)
      begin
        if Error  nil then
        begin
          var lError := EException.CreateFmt("Failed to create device [%s] with path: %s", [DiskName, lFullPath]);
          if assigned(CB) then
            CB(TagValue, PathName, lError)
          else
            raise lError;
          exit;
        end;

        FDatabase.Execute(
          #'insert into devices (type, owner, name, location)
            values(?, ?, ?, ?);',
            [ord(JDeviceType.dtLocal), UserName, Diskname, PathName] ,
        procedure (Sender: TObject; Error: Exception)
        begin
          if Error  nil then
          begin
            WriteToLog(Error.message);
            if assigned(CB) then
              CB(TagValue, PathName, Error)
            else
              raise Error;
            exit;
          end;

          WriteToLogF("Device [%s] added to database user [%s]", [DiskName, UserName]);
          if assigned(CB) then
            CB(TagValue, PathName, nil);
        end);

      end);



    end);
  end);
end;

procedure TQTXTaskService.SetupDeviceTable(const TagValue: variant; const CB: TRagnarokServiceCB);
begin

  FDatabase.Execute(
    #'
      create table if not exists devices
          (
            id integer primary key AUTOINCREMENT,
            type        integer,
            owner       text,
            name        text,
            location    text,
            apikey      text,
            apisecret   text,
            apipassword text,
            apiuser     text
          );
          ', [],
    procedure (Sender: TObject; Error: Exception)
    begin
      if Error  nil then
      begin
        WriteToLog(Error.message);
        if assigned(CB) then
          CB(Error)
        else
          raise Error;
        exit;
      end else
      if assigned(CB) then
        CB(nil);
    end);
end;

procedure TQTXTaskService.SetupTaskTable(const TagValue: variant; const CB: TRagnarokServiceCB);
begin

  FDatabase.Execute(
    #'
      create table if not exists tasks
          (
            id integer primary key AUTOINCREMENT,
            state     integer,
            username  text,
            created   real
          );
          ', [],
    procedure (Sender: TObject; Error: Exception)
    begin
      if Error  nil then
      begin
        WriteToLog(Error.message);
        if assigned(CB) then
          CB(Error)
        else
          raise Error;
        exit;
      end else
      if assigned(CB) then
        CB(nil);
    end);
end;


procedure TQTXTaskService.SetupOperationsTable(const TagValue: variant; const CB: TRagnarokServiceCB);
begin
  FDatabase.Execute(
    #'
      create table if not exists operations
          (
            id integer primary key AUTOINCREMENT,
            username text,
            taskid integer,
            name text,
            filename text
          );
          ', [],
    procedure (Sender: TObject; Error: Exception)
    begin
      if Error  nil then
      begin
        WriteToLog(Error.message);
        if assigned(CB) then
          CB(Error)
        else
          raise Error;
        exit;
      end else
      if assigned(CB) then
        CB(nil);
    end);
end;

procedure TQTXTaskService.SetupDatabase(const CB: TRagnarokServiceCB);
begin
  // Try to read database-path from preferences file
  var LDbFileToOpen := FPrefs.ReadString("database", "database_name", "");

  // Trim away spaces, check if there is a filename
  LDbFileToOpen := LDbFileToOpen.trim();
  if LDbFileToOpen.length < 1 then
  begin
    // No filename? Fall back on pre-defined file in CWD
    var LBasePath := TQTXNodeFileUtils.GetCurrentDirectory();
    LDbFileToOpen := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(LBasePath) + CNT_PREFS_DBNAME;
  end;

  FDatabase.AccessMode := TSQLite3AccessMode.sqaReadWriteCreate;
  FDatabase.Open(LDbFileToOpen,
    procedure (Sender: TObject; Error: Exception)
    begin
      if Error  nil then
      begin
        WriteToLog(Error.message);
        if assigned(CB) then
          CB(Error)
        else
          raise Error;
        exit;
      end;

      WriteToLog("Initializing task table");
      SetupTaskTable(nil, procedure (Error: exception)
      begin
        if Error  nil then
        begin
          WriteToLog("Tasks initialized: **failed");
          WriteToLog(error.message);
          if assigned(CB) then
            CB(Error)
          else
            raise Error;
          exit;
        end else
        writeToLog("Tasks initialized: OK");

        WriteToLog("Initializing operations table");
        SetupOperationsTable(nil, procedure (Error: exception)
        begin
          if Error  nil then
          begin
            WriteToLog("Operations initialized: **failed");
            WriteToLog(error.message);
            if assigned(CB) then
              CB(Error);
            exit;
          end else
          writeToLog("Operations initialized: OK");

          WriteToLog("Initializing device table");
          SetupDeviceTable(nil, procedure (Error: exception)
          begin
            if Error  nil then
            begin
              WriteToLog("Device-table initialized: **failed");
              WriteToLog(error.message);
              if assigned(CB) then
                CB(Error);
              exit;
            end else
            writeToLog("Device-table initialized: OK");

            if assigned(CB) then
              CB(nil);
          end);
        end);
      end);
    end);
end;


procedure TQTXTaskService.HandleFileRead(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
begin
  var lRequest := TQTXFileReadRequest(request);
  var lUserName := lRequest.UserName;
  var lFileName := lRequest.FileName;

  // Check filename length
  if lFileName.length  0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  lTemp := './';
  if pos(lTemp, lFileName) > 0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  GetFileInfo(lRequest, lUserName, lFileName,
  procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception)
  begin
    if Error  nil then
    begin
      WriteToLog(Error.message);
      SendError(Socket, Request, Error.Message);
      exit;
    end;

    var lOptions: TReadFileOptions;
    lOptions.encoding := 'binary';

    NodeJsFsAPI().readFile(LocalFile, lOptions,
    procedure (Error: JError; Data: JNodeBuffer)
    begin
      if Error  nil then
      begin
        WriteToLog(Error.message);
        SendError(Socket, Request, Error.Message);
        exit;
      end;

      var lResponse := TQTXFileReadResponse.Create(Request.Ticket);
      lResponse.UserName := lUserName;
      lResponse.Routing.TagValue := request.routing.tagValue;
      lResponse.FileName := lFileName;
      lResponse.Code := CNT_MESSAGE_CODE_OK;
      lResponse.Response := CNT_MESSAGE_TEXT_OK;

      // Convert filedata in one pass
      try
        var lConvert := TDataTypeConverter.Create();
        try
          lResponse.Attachment.AppendBytes( lConvert.TypedArrayToBytes(Data) );
        finally
          lConvert.free;
        end;
      except
        on e: exception do
        begin
          WriteToLog(e.message);
          SendError(Socket, Request, e.Message);
          exit;
        end;
      end;

      try
        Socket.Send( lResponse.Serialize() );
      except
        on e: exception do
          WriteToLog(e.message);
      end;
    end);
  end);
end;

procedure TQTXTaskService.HandleFileReadPartial(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
begin
  var lRequest := TQTXFileReadPartialRequest(request);
  var lUserName := lRequest.UserName;
  var lFileName := lRequest.FileName;
  var lStart := lRequest.Offset;
  var lSize := lRequest.Size;

  // Check filename length
  if lFileName.length  0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  lTemp := './';
  if pos(lTemp, lFileName) > 0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  if lSize < 1 then
  begin
    SendError(Socket, Request, "Read failed, invalid size error");
    exit;
  end;

  if lStart < 0 then
  begin
    SendError(Socket, Request, "Read failed, invalid offset error");
    exit;
  end;

  GetFileInfo(lRequest, lUserName, lFileName,
  procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception)
  begin
    if Error  nil then
    begin
      WriteToLog(Error.message);
      SendError(Socket, Request, Error.Message);
      exit;
    end;

    if lStart > Info.size then
    begin
      SendError(Socket, Request, "Read failed, offset beyond filesize error");
      exit;
    end;

    NodeJsFsAPI().open(LocalFile, "r",
    procedure (Error: JError; Fd: THandle)
    begin
      if error  nil then
      begin
        WriteToLog(Error.message);
        SendError(Socket, Request, Error.Message);
        exit;
      end;

      var Data = new JNodeBuffer(lSize);
      NodeJsFsAPI().read(Fd, Data, 0, lSize, lStart,
      procedure (Error: JError; BytesRead: integer; buffer: JNodeBuffer)
      begin
        if Error  nil then
        begin
          NodeJsFsAPI().closeSync(Fd);
          WriteToLog(Error.message);
          SendError(Socket, Request, Error.Message);
          exit;
        end;

        // Close the file-handle and return data
        NodeJsFsAPI().close(Fd, procedure (Error: JError)
        begin
          var lResponse := TQTXFileReadPartialResponse.Create(Request.Ticket);
          lResponse.UserName := lUserName;
          lResponse.Routing.TagValue := request.routing.tagValue;
          lResponse.FileName := lFileName;
          lResponse.Code := CNT_MESSAGE_CODE_OK;
          lResponse.Response := CNT_MESSAGE_TEXT_OK;

          // Only encode data if read
          if BytesRead > 0 then
          begin
            // Convert filedata in one pass
            try
              var lConvert := TDataTypeConverter.Create();
              try
                lResponse.Attachment.AppendBytes( lConvert.TypedArrayToBytes(buffer) );
              finally
                lConvert.free;
              end;
            except
              on e: exception do
              begin
                WriteToLog(e.message);
                SendError(Socket, Request, e.Message);
                exit;
              end;
            end;
          end;

          try
            Socket.Send( lResponse.Serialize() );
          except
            on e: exception do
              WriteToLog(e.message);
          end;

        end);
      end);
    end);
  end);
end;

procedure TQTXTaskService.HandleFileWrite(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
begin
  var lRequest  := TQTXFileWriteRequest(request);
  var lFileName := lRequest.FileName.trim();
  var lUserName := lRequest.UserName.trim();

  var FullPath  := lFileName;

  // Check filename length
  if lFileName.length  0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  lTemp := './';
  if pos(lTemp, lFileName) > 0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  var lParser := TQTXPathParser.Create();
  try
    var lInfo: TQTXPathData;
    if lparser.Parse(FullPath, lInfo) then
    begin
      // Locate the device for the path belonging to the user
      FindDeviceByName(nil, lUserName, lInfo.MountPart,
      procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception)
      begin
        if Error  nil then
        begin
          WriteToLog(Error.Message);
          SendError(Socket, Request, Error.Message);
          exit;
        end;

        case Device.&Type of
        dtLocal:
          begin
            // We want the path + filename, so we can append that to
            // the actual localized filesystem
            var lExtract := FullPath;
            delete(lExtract, 1, lInfo.MountPart.Length + 1);

            // Construct complete storage location
            var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory();
            lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices';
            lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim();
            lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + lExtract;

            // Extract data to be appended, if any
            // note: null bytes should be allowed, it should just create the file
            var lBytes: array of UInt8;
            if lRequest.attachment.Size > 0 then
              lBytes := lRequest.Attachment.ToBytes();

            // Write the data to the file
            NodeJsFsAPI().writeFile(lFullPath, lBytes,
            procedure (Error: JError)
            begin
              if Error  nil then
              begin
                WriteToLog(Error.Message);
                SendError(Socket, Request, Error.Message);
                exit;
              end;

              // Setup response object
              var lResponse := TQTXFileWriteResponse.Create(lRequest.Ticket);
              lResponse.UserName := lUserName;
              lResponse.FileName := lFileName;
              lResponse.Code := CNT_MESSAGE_CODE_OK;
              lResponse.Response := CNT_MESSAGE_TEXT_OK;

              // Send success response
              try
                Socket.Send( lResponse.Serialize() );
              except
                on e: exception do
                  WriteToLog(e.message);
              end;

            end);

          end;
        dtDropbox, dtGoogle, dtMsDrive:
          begin
            var lErrorText := Format("Clound bindings not active error [%s]", [lRequest.FileName]);
            WriteToLog(lErrorText);
            SendError(Socket, Request, lErrorText);
          end;
        end;
      end);
    end else
    begin
      SendError(Socket, Request, format("Invalid path [%s] error", [FullPath]));
    end;
  finally
    lParser.free;
  end;
end;

procedure TQTXTaskService.HandleFileWritePartial(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
begin
  var lRequest  := TQTXFileWritePartialRequest(request);
  var lFileName  := lRequest.FileName.trim();
  var lUserName := lRequest.UserName.trim();
  var lFileOffset := lRequest.Offset;

  // Check filename length
  if lFileName.length  0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  lTemp := './';
  if pos(lTemp, lFileName) > 0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  var FullPath := lFileName;

  var lParser := TQTXPathParser.Create();
  try
    var lInfo: TQTXPathData;
    if lparser.Parse(FullPath, lInfo) then
    begin
      // Locate the device for the path belonging to the user
      FindDeviceByName(nil, lUserName, lInfo.MountPart,
      procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception)
      begin
        if Error  nil then
        begin
          WriteToLog(Error.Message);
          SendError(Socket, Request, Error.Message);
          exit;
        end;

        case Device.&Type of
        dtLocal:
          begin
            // We want the path + filename, so we can append that to
            // the actual localized filesystem
            var lExtract := FullPath;
            delete(lExtract, 1, lInfo.MountPart.Length + 1);

            // Construct complete storage location
            var lFullPath := TQTXNodeFileUtils.GetCurrentDirectory();
            lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + 'userdevices';
            lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + Device.location.trim();
            lFullPath := TQTXNodeFileUtils.IncludeTrailingPathDelimiter(lFullPath) + lExtract;

            // Extract data to be appended, if any
            // note: null bytes should be allowed, it should just create the file
            var lBytes: array of UInt8;
            if lRequest.attachment.Size > 0 then
              lBytes := lRequest.Attachment.ToBytes();

            var lAccess := TQTXNodeFile.Create();
            lAccess.Open(lFullPath, TQTXNodeFileMode.nfWrite,
            procedure (Error: Exception)
            begin
              if Error  nil then
              begin
                WriteToLog(Error.Message);
                SendError(Socket, Request, Error.Message);
                exit;
              end;

              lAccess.Write(lBytes, lFileOffset,
              procedure (Error: Exception)
              begin
                if Error  nil then
                begin
                  WriteToLog(Error.Message);
                  SendError(Socket, Request, Error.Message);
                  exit;
                end;

                // Setup response object
                var lResponse := TQTXFileWriteResponse.Create(lRequest.Ticket);
                lResponse.UserName := lUserName;
                lResponse.FileName := lFileName;
                lResponse.Code := CNT_MESSAGE_CODE_OK;
                lResponse.Response := CNT_MESSAGE_TEXT_OK;

                // Send success response
                try
                  Socket.Send( lResponse.Serialize() );
                except
                  on e: exception do
                    WriteToLog(e.message);
                end;

              end);
            end);
          end;
        dtDropbox, dtGoogle, dtMsDrive:
          begin
            var lErrorText := Format("Clound bindings not active error [%s]", [lRequest.FileName]);
            WriteToLog(lErrorText);
            SendError(Socket, Request, lErrorText);
          end;
        end;
      end);
    end else
    begin
      SendError(Socket, Request, format("Invalid path [%s] error", [FullPath]));
    end;
  finally
    lParser.free;
  end;
end;

procedure TQTXTaskService.HandleRmDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
begin
  var lRequest := TQTXRmDirRequest(request);
  var lUserName := lRequest.UserName.trim();
  var lDirPath := lRequest.DirPath.trim();

  if lDirPath.length  0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  lTemp := './';
  if pos(lTemp, lDirPath) > 0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  var lParser := TQTXPathParser.Create();
  try
    var lInfo: TQTXPathData;
    if lParser.Parse(lDirPath, lInfo) then
    begin
      GetTranslatedPathFor(nil, lUserName, lDirPath,
      procedure (TagValue: variant; Original, Translated: string; Error: Exception)
      begin
        if Error  nil then
        begin
          WriteToLog(Error.message);
          SendError(Socket, Request, Error.Message);
          exit;
        end;

        if not TQTXNodeFileUtils.DirectoryExists(Translated) then
        begin
          WriteToLogF("RmDir Failed, directory [%s] does not exist", [Translated]);
          SendError(Socket, Request, Format("RmDir failed, directory [%s] does not exist", [Original]));
          exit;
        end;

        TQTXNodeFileUtils.DeleteDirectory(nil, Translated,
        procedure (TagValue: variant; Path: string; Error: Exception)
        begin
          if error  nil then
          begin
            WriteToLog(Error.message);
            SendError(Socket, Request, Error.Message);
            exit;
          end;

          // Setup response object
          var lResponse := TQTXRmDirResponse.Create(lRequest.Ticket);
          lResponse.UserName := lUserName;
          lResponse.DirPath := lDirPath;
          lResponse.Code := CNT_MESSAGE_CODE_OK;
          lResponse.Response := CNT_MESSAGE_TEXT_OK;
          lResponse.Routing.TagValue := lRequest.Routing.TagValue;

          // Send success response
          try
            Socket.Send( lResponse.Serialize() );
          except
            on e: exception do
              WriteToLog(e.message);
          end;
        end);
      end);
    end else
    begin
      var lText := format("RmDir failed, invalid path [%s] error", [lDirPath]);
      WriteToLog(lText);
      SendError(Socket, Request, lText);
    end;
  finally
    lParser.free;
  end;
end;

procedure TQTXTaskService.HandleMkDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
begin
  var lRequest := TQTXMkDirRequest(request);
  var lUserName := lRequest.UserName.trim();
  var lDirPath := lRequest.DirPath.trim();

  if lDirPath.length  0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  lTemp := './';
  if pos(lTemp, lDirPath) > 0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  var lParser := TQTXPathParser.Create();
  try
    var lInfo: TQTXPathData;
    if lparser.Parse(lDirPath, lInfo) then
    begin
      GetTranslatedPathFor(nil, lUserName, lDirPath,
      procedure (TagValue: variant; Original, Translated: string; Error: Exception)
      begin
        if Error  nil then
        begin
          WriteToLog(Error.message);
          SendError(Socket, Request, Error.Message);
          exit;
        end;

        TQTXNodeFileUtils.DirectoryExists(nil, Translated,
        procedure (TagValue: variant; Path: string; Error: Exception)
        begin
          if Error  nil then
          begin
            WriteToLogF("MkDir Failed, directory [%s] already exists", [Translated]);
            SendError(Socket, Request, Format("MkDir Failed, directory [%s] already exists", [Original]));
            exit;
          end;

          TQTXNodeFileUtils.CreateDirectory(nil, Translated,
          procedure (TagValue: variant; Path: string; Error: Exception)
          begin
            if Error  nil then
            begin
              WriteToLogF("MkDir Failed, directory [%s] could not be created", [Original]);
              SendError(Socket, Request, Format("MkDir Failed, directory [%s] could not be created", [Translated]));
              exit;
            end;

            // Setup response object
            var lResponse := TQTXMkDirResponse.Create(lRequest.Ticket);
            lResponse.UserName := lUserName;
            lResponse.DirPath := lDirPath;
            lResponse.Code := CNT_MESSAGE_CODE_OK;
            lResponse.Response := CNT_MESSAGE_TEXT_OK;
            lResponse.Routing.TagValue := lRequest.Routing.TagValue;

            // Send success response
            try
              Socket.Send( lResponse.Serialize() );
            except
              on e: exception do
                WriteToLog(e.message);
            end;

          end);
        end);
      end);

    end else
    begin
      var lText := format("MkDir Failed, invalid path [%s] error", [lDirPath]);
      WriteToLog(lText);
      SendError(Socket, Request, lText);
    end;
  finally
    lParser.free;
  end;
end;

procedure TQTXTaskService.HandleFileDelete(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
begin
  var lRequest := TQTXFileDeleteRequest(Request);
  var lUserName := lRequest.UserName.trim();
  var lFileName := lRequest.FileName.trim();

  if lFileName.length  0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  lTemp := './';
  if pos(lTemp, lFileName) > 0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  GetFileInfo(lRequest, lUserName, lFileName,
  procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception)
  begin
    if Error  nil then
    begin
      WriteToLog(Error.message);
      SendError(Socket, Request, Error.Message);
      exit;
    end;

    if not Info.isFile then
    begin
      SendError(Socket, Request, "Filesystem object is not a file error");
      exit;
    end;

    NodeJsFsAPI().unlink(LocalFile,
    procedure (Error: JError)
    begin
      if Error  nil then
      begin
        WriteToLog(Error.message);
        SendError(Socket, Request, Error.message);
        exit;
      end;

      var lResponse := new TQTXFileDeleteResponse(lRequest.Ticket);
      lResponse.Routing.TagValue := request.Routing.TagValue;
      lResponse.UserName := lUserName;
      lResponse.FileName := lFileName;
      lResponse.Code := CNT_MESSAGE_CODE_OK;
      lResponse.Response := CNT_MESSAGE_TEXT_OK;

      try
        Socket.Send( lResponse.Serialize() );
      except
        on e: exception do
          WriteToLog(e.message);
      end;
    end);
  end);
end;

procedure TQTXTaskService.HandleFileRename(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
begin
  var lRequest := TQTXFileRenameRequest(Request);
  var lUserName := lRequest.UserName.trim();
  var lFileName := lRequest.FileName.trim();
  var lNewName := lRequest.NewName.trim();

  // Check filename length
  if lFileName.length < 1 then
  begin
    SendError(Socket, Request, Format("Invalid or empty from-filename [%s] error", [lFileName]) );
    exit;
  end;

  // check newname length
  if lNewName.length  0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  if pos(lTemp, lNewName) > 0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  lTemp := './';
  if pos(lTemp, lFileName) > 0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  if pos(lTemp, lNewName) > 0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;


  GetFileInfo(lRequest, lUserName, lFileName,
  procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception)
  begin
    if Error  nil then
    begin
      WriteToLog(Error.message);
      SendError(Socket, Request, Error.Message);
      exit;
    end;

    if not Info.isFile then
    begin
      SendError(Socket, Request, "Filesystem object is not a file error");
      exit;
    end;

    GetTranslatedPathFor(nil, lUsername, lNewName,
    procedure (TagValue: variant; Original, Translated: string; Error: Exception)
    begin
      if Error  nil then
      begin
        WriteToLog(Error.message);
        SendError(Socket, Request, Error.Message);
        exit;
      end;

      NodeJsFsAPI().rename(LocalFile, Translated,
      procedure (Error: JError)
      begin
        if Error  nil then
        begin
          WriteToLog(Error.message);
          SendError(Socket, Request, Error.message);
          exit;
        end;

        var lResponse := new TQTXFileRenameResponse(lRequest.Ticket);
        lResponse.Routing.TagValue := request.Routing.TagValue;
        lResponse.UserName := lUserName;
        lResponse.FileName := lFileName;
        lResponse.Code := CNT_MESSAGE_CODE_OK;
        lResponse.Response := CNT_MESSAGE_TEXT_OK;

        try
          Socket.Send( lResponse.Serialize() );
        except
          on e: exception do
            WriteToLog(e.message);
        end;
      end);

    end);

  end);
end;

procedure TQTXTaskService.HandleGetDir(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
begin
  var lRequest := TQTXFileDirRequest(Request);
  var lUserName := lRequest.UserName.trim();
  var lPath := lRequest.Path.trim();

  // prevent path escape attempts
  var lTemp := "../";
  if pos(lTemp, lPath) > 0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  lTemp := './';
  if pos(lTemp, lPath) > 0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  GetTranslatedPathFor(nil, lUserName, lPath,
  procedure (TagValue: variant; Original, Translated: string; Error: Exception)
  begin
    if Error  nil then
    begin
      WriteToLog(Error.message);
      SendError(Socket, Request, Error.Message);
      exit;
    end;

    //writeln("Translated path is:" + Translated);

    if not TQTXNodeFileUtils.DirectoryExists(Translated) then
    begin
      WriteToLogF("GetDir Failed, directory [%s] does not exist", [Translated]);
      SendError(Socket, Request, Format("GetDir failed, directory [%s] does not exist", [Original]));
      exit;
    end;

    var lWalker := TQTXFileWalker.Create();
    lWalker.Examine(Translated, procedure (Sender: TQTXFileWalker; Error: EException)
    begin
      if Error  nil then
      begin
        WriteToLogF("GetDir Failed: %s", [Error.Message]);
        SendError(Socket, Request, Format("GetDir failed: %s", [Error.Message]));
        exit;
      end;

      // Get the directory data, swap out the path
      // record with the original [amiga] style path
      var lData := Sender.ExtractList();
      lData.dlPath := Original;

      var lResponse := new TQTXFileDirResponse(lRequest.Ticket);
      lResponse.Routing.TagValue := request.Routing.TagValue;
      lResponse.UserName := lUserName;
      lResponse.Path := lPath;
      lResponse.Assign( lData );

      try
        Socket.Send( lResponse.Serialize() );
      except
        on e: exception do
          WriteToLog(e.message);
      end;

      // release instance in 100ms
      TQTXDispatch.execute(procedure ()
      begin
        try
          lWalker.free
        except
          on e: exception do
          begin
            WriteToLogF("Failed to release file-walker instance: %s", [e.message]);
          end;
        end;
      end, 100);
    end);
  end);
end;

procedure TQTXTaskService.HandleGetFileInfo(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
begin
  var lRequest := TQTXFileInfoRequest(Request);
  var lUserName := lRequest.UserName.trim();
  var lFileName := lRequest.FileName.trim();

  // prevent path escape attempts
  var lTemp := "../";
  if pos(lTemp, lFileName) > 0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  lTemp := './';
  if pos(lTemp, lFileName) > 0 then
  begin
    SendError(Socket, Request, Format("Unsupported path sequence [%s] detected error", [lTemp]) );
    exit;
  end;

  GetFileInfo(lRequest, lUserName, lFileName,
  procedure (TagValue: variant; LocalFile: string; Info: JStats; Error: Exception)
  begin
    if Error  nil then
    begin
      WriteToLog(Error.message);
      SendError(Socket, Request, Error.Message);
      exit;
    end;

    // Collect the data
    var lData := new JFileItem();
    lData.diFileName := lFileName;
    lData.diFileType := if Info.isFile then JFileItemType.wtFile else JFileItemType.wtFolder;
    lData.diFileSize := Info.size;
    lData.diFileMode := IntToStr(Info.mode);
    lData.diCreated  := TDateUtils.FromJsDate( Info.cTime );
    lData.diModified := TDateUtils.FromJsDate( Info.mTime );

    var lResponse := new TQTXFileInfoResponse(lRequest.Ticket);
    lResponse.Routing.TagValue := request.Routing.TagValue;
    lResponse.UserName := lUserName;
    lResponse.FileName := lFileName;
    lResponse.Assign(lData);

    try
      Socket.Send( lResponse.Serialize() );
    except
      on e: exception do
        WriteToLog(e.message);
    end;
  end);
end;

procedure TQTXTaskService.HandleDestroyDevice(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
begin
  var lMessage := TQTXFileDestroyDeviceRequest(request);

  // This will also destroy any files + unregister the device in the
  // database table for the service -- do not mess with this!
  UnRegisterLocalDevice(nil, lMessage.Username, lMessage.DeviceName,
  procedure (TagValue: variant; LocalPath: string; Error: Exception)
  begin
    if Error  nil then
    begin
      WriteToLog(Error.Message);
      SendError(Socket, Request, Error.Message);
      exit;
    end;

    var lResponse := TQTXFileDestroyDeviceResponse.Create(request.ticket);
    lResponse.UserName := lMessage.UserName;
    lResponse.DeviceName := lMessage.DeviceName;
    lResponse.Routing.TagValue := Request.Routing.TagValue;
    lResponse.Code := CNT_MESSAGE_CODE_OK;
    lResponse.Response := CNT_MESSAGE_TEXT_OK;

    try
      Socket.Send( lResponse.Serialize() );
    except
      on e: exception do
      begin
        WriteToLog(e.message);
      end;
    end;
  end);
end;

procedure TQTXTaskService.HandleCreateLocalDevice(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
begin
  var lMessage := TQTXFileCreateLocalDeviceRequest(request);

  // Attempt to register.
  // NOTE: This will automatically create a matching folder
  //       under $cwd/userdevices/[calculated_name_of_device]

  RegisterLocalDevice(nil, lMessage.Username, lMessage.DeviceName,
  procedure (TagValue: variant; LocalPath: string; Error: Exception)
  begin
    if Error  nil then
    begin
      WriteToLog(Error.Message);
      SendError(Socket, Request, Error.Message);
      exit;
    end;

    FindDeviceByName(nil, lMessage.Username, lMessage.DeviceName,
    procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception)
    begin
      if Error  nil then
      begin
        WriteToLog(Error.Message);
        SendError(Socket, Request, Error.Message);
        exit;
      end;

      var lResponse := TQTXFileCreateLocalDeviceResponse.Create(request.ticket);
      lResponse.UserName := lMessage.UserName;
      lResponse.Routing.TagValue := Request.Routing.TagValue;
      lResponse.Code := CNT_MESSAGE_CODE_OK;
      lResponse.Response := CNT_MESSAGE_TEXT_OK;
      if Device  nil then
        lResponse.assign(Device);

      try
        Socket.Send( lResponse.Serialize() );
      except
        on e: exception do
        begin
          WriteToLog(e.message);
        end;
      end;

    end);
  end);
end;

procedure TQTXTaskService.HandleGetDeviceByName(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
begin
  var lMessage := TQTXFileGetDeviceByNameRequest(request);

  FindDeviceByName(nil, lMessage.Username, lMessage.DeviceName,
  procedure (TagValue: variant; Device: JDeviceInfo; Error: Exception)
  begin
    if Error  nil then
    begin
      WriteToLog(Error.Message);
      SendError(Socket, Request, Error.Message);
      exit;
    end;

    var lResponse := TQTXFileGetDeviceByNameResponse.Create(request.ticket);
    lResponse.UserName := lMessage.UserName;
    lResponse.Code := CNT_MESSAGE_CODE_OK;
    lResponse.Response := CNT_MESSAGE_TEXT_OK;
    if Device  nil then
      lResponse.assign(Device);

    try
      Socket.Send( lResponse.Serialize() );
    except
      on e: exception do
      begin
        WriteToLog(e.message);
      end;
    end;
  end);

end;

procedure TQTXTaskService.HandleGetDevices(Socket: TNJWebSocketSocket; Request: TQTXBaseMessage);
begin
  var lMessage := TQTXFileGetDeviceListRequest(Request);
  GetDevicesForUser(nil, lMessage.Username,
  procedure (TagValue: variant; Devices: JDeviceList; Error: Exception)
  begin
    if Error  nil then
    begin
      WriteToLog(Error.Message);
      SendError(Socket, Request, Error.Message);
      exit;
    end;

    var lResponse := TQTXFileGetDeviceListResponse.Create(request.ticket);
    lResponse.UserName := lMessage.UserName;
    lResponse.Code := CNT_MESSAGE_CODE_OK;
    lResponse.Response := CNT_MESSAGE_TEXT_OK;
    if Devices  nil then
      lResponse.assign(Devices);

    try
      Socket.Send( lResponse.Serialize() );
    except
      on e: exception do
      begin
        WriteToLog(e.message);
      end;
    end;

  end);
end;

procedure TQTXTaskService.AfterServerStarted;
begin
  inherited;

  // Check prefs if zconfig should be applied
  if self.FPrefs.ReadBoolean("zconfig", "active", false) then
  begin
    // ZConfig should only run on the master instance.
    // We dont want to register our endpoint for each worker
    if NodeJSClusterAPI().isWorker then
      exit;

    writeln("Setting up Zero-Configuration layer");
    FZConfig.port := FPrefs.ReadInteger('zconfig', 'bindport', 2109);
    FZConfig.address := GetMachineIP();
    FZConfig.Start(nil, procedure (Sender: TObject; TagValue: variant; Error: Exception)
    begin
      if FPrefs.ReadBoolean("zconfig", "broadcast", true) then
        FZConfig.Socket.setBroadcast(true);

      // Build up the endpoint (URL) for our websocket server
      var lEndpoint := '';

      if FPrefs.ReadBoolean('networking', 'secure', false) then
        lEndpoint := 'wss://'
      else
        lEndpoint := 'ws://';

      lEndpoint += GetMachineIP();
      lEndpoint += ':' + Port.ToString();

      // Ping the ZConfig service on interval, until our service is registered
      // We keep track of the interval handle so we can stop calling on interval later
      FRegHandle := TQTXDispatch.SetInterval( procedure ()
      begin
        inc(FRegCount);

        // Only output once to avoid overkill in the log
        if FRegCount = 1 then
          WriteToLogF("ZConfig registration begins [%s]", [lEndpoint]);

        FZConfig.RegisterService(nil, CNT_ZCONFIG_SERVICE_NAME, SERVICE_ID_TASKMANAGER, lEndpoint,
        procedure (TagValue: variant; Error: Exception)
        begin
          if Error = nil then
          begin
            WriteToLog("Service registered");
            TQTXDispatch.ClearInterval(FRegHandle);
            FRegCount := 0;
            exit;
          end;
        end);
      end, 1000);

    end);
  end;
end;

procedure TQTXTaskService.BeforeServerStopped;
begin
  inherited;
end;

procedure TQTXTaskService.Dispatch(Socket: TNJWebSocketSocket; Message: TQTXBaseMessage);
begin
  var LInfo := MessageDispatch.GetMessageInfoForClass(Message);
  if LInfo  nil then
  begin
    try
      LInfo.MessageHandler(Socket, Message);
    except
      on e: exception do
      begin
        //Log error
        WriteToLog(e.message);
      end;
    end;
  end;
end;

end.


 

Quartex “Cloud Ripper” hardware

November 10, 2019 Leave a comment

For close to a year now I have been busy on a very exciting project, namely my own cloud system. While I have written about this project quite a bit these past months, mostly focusing on the software aspect, not much has been said about that hardware.

74238389_10156646805205906_1728576808808349696_o

Quartex “Cloud Ripper” running neatly on my home-office desk

So let’s have a look at Cloud Ripper, the official hardware setup for Quartex Media Desktop.

Tiny footprint, maximum power

Despite its complexity, the Quartex Media Desktop architecture is surprisingly lightweight. The services that makes up the baseline system (read: essential services) barely consume 40 megabytes of ram per instance (!). And while there is a lot of activity going on between these services -most of that activity is message-dispatching. Sending messages costs practically nothing in cpu and network terms. This will naturally change the moment you run your cloud as a public service, or setup the system in an office environment for a team. The more users, the more signals are shipped between the processes – but with the exception of reading and writing large files, messages are delivered practically instantaneous and hardly use CPU time.

CloudRipper

Quartex Media Desktop is based on a clustered micro-service architecture

One of the reasons I compile my code to JavaScript (Quartex Media Desktop is written from the ground up in Object Pascal, which is compiled to JavaScript) has to do with the speed and universality of node.js services. As you might know, Node.js is powered by the Google V8 runtime engine, which means the code is first converted to bytecodes, and further compiled into highly optimized machine-code [courtesy of llvm]. When coded right, such Javascript based services execute just as fast as those implemented in a native language. There simply are no perks to be gained from using a native language for this type of work. There are however plenty of perks from using Node.js as a service-host:

  • Node.js delivers the exact same behavior no matter what hardware or operating-system you are booting up from. In our case we use a minimal Linux setup with just enough infrastructure to run our services. But you can use any OS that supports Node.js. I actually have it installed on my Android based Smart-TV (!)
  • We can literally copy our services between different machines and operating systems without recompiling a line of code. So we don’t need to maintain several versions of the same software for different systems.
  • We can generate scripts “on the fly”, physically ship the code over the network, and execute it on any of the machines in our cluster. While possible to do with native code, it’s not very practical and would raise some major security concerns.
  • Node.js supports WebAssembly, you can use the Elements Compiler from RemObjects to write service modules that executes blazingly fast yet remain platform and chipset independent.

The Cloud-Ripper cube

The principal design goal when I started the project, was that it should be a distributed system. This means that instead of having one large-service that does everything (read: a typical “native” monolithic design), we instead operate with a microservice cluster design. Services that run on separate SBC’s (single board computers). The idea here is to spread the payload over multiple mico-computers that combined becomes more than the sum of their parts.

IMG_4644_Product_1024x1024@2x

Cloud Ripper – Based on the Pico 5H case and fitted with 5 x ODroid XU4 SBC’s

So instead of buying a single, dedicated x86 PC to host Quartex Media Desktop, you can instead buy cheap, off-the-shelves, easily available single-board computers and daisy chain them together. So instead of spending $800 (just to pin a number) on x86 hardware, you can pick up $400 worth of cheap ARM boards and get better network throughput and identical processing power (*). In fact, since Node.js is universal you can mix and match between x86, ARM, Mips and PPC as you see fit. Got an older PPC Mac-Mini collecting dust? Install Linux on it and get a few extra years out of these old gems.

(*) A single XU4 is hopelessly underpowered compared to an Intel i5 or i7 based PC. But in a cluster design there are more factors than just raw computational power. Each board has 8 CPU cores, bringing the total number of cores to 40. You also get 5 ARM Mali-T628 MP6 GPUs running at 533MHz. Only one of these will be used to render the HTML5 display, leaving 4 GPUs available for video processing, machine learning or compute tasks. Obviously these GPUs won’t hold a candle to even a mid-range graphics card, but the fact that we can use these chips for audio, video and computation tasks makes the system incredibly versatile.

Another design goal was to implement a UDP based Zero-Configuration mechanism. This means that the services will find and register with the core (read: master service) automatically, providing the machines are all connected to the same router or switch.

IMG_4650_Product_1024x1024@2x

Put together your own supercomputer for less than $500

The first “official” hardware setup is a cluster based on 5 cheap ARM boards; namely the ODroid XU4. The entire setup fits inside a Pico Cube, which is a special case designed to house this particular model of single board computers. Pico offers several different designs, ranging from 3 boards to a 20 board super-cluster. You are not limited ODroid XU4 boards if you prefer something else. I picked the XU4 boards because they represent the lowest possible specs you can run the Quartex Media Desktop on. While the services themselves require very little, the master board (the board that runs the QTXCore.js service) is also in charge of rendering the HTML5 display. And having tested a plethora of boards, the ODroid XU4 was the only model that could render the desktop properly (at that low a price range).

Note: If you are thinking about using a Raspberry PI 3B (or older) as the master SBC, you can pretty much forget it. The media desktop is a piece of very complex HTML5, and anything below an ODroid XU4 will only give you a terrible experience (!). You can use smaller boards as slaves, meaning that they can host one of the services, but the master should preferably be an ODroid XU4 or better. The ODroid N2 [with 4Gb Ram] is a much better candidate than a Raspberry PI v4. A Jetson Nano is an even better option due to its extremely powerful GPU.

Booting into the desktop

One of the things that confuse people when they read about the desktop project, is how it’s possible to boot into the desktop itself and use Quartex Media Desktop as a ChromeOS alternative?

How can a “cloud platform” be used as a desktop alternative? Don’t you need access to the internet at all times? If it’s a server based system, how then can we boot into it? Don’t we need a second PC with a browser to show the desktop?

73475069_10156646805615906_2668445017588105216_o

Accessing the desktop like a “web-page” from a normal Linux setup

To make a long story short: the “master” in our cluster architecture (read: the single-board computer defined as the boss) is setup to boot into a Chrome browser display under “kiosk mode”. When you start Chrome in kiosk mode, this removes all traces of the ordinary browser experience. There will be no toolbars, no URL field, no keyboard shortcuts, no right-click popup menus etc. It simply starts in full-screen and whatever HTML5 you load, has complete control over the display.

What I have done, is to to setup a minimal Linux boot sequence. It contains just enough Linux to run Chrome. So it has all the drivers etc. for the device, but instead of starting the ordinary Linux Desktop (X or Wayland) -we instead start Chrome in kiosk mode.

74602781_10156646805300906_6294526665393438720_o

Booting into the same desktop through Chrome in Kiosk Mode. In this mode, no Linux desktop is required. The Linux boot sequence is altered to jump straight into Chrome

Chrome is started to load from 127.0.0.1 (this is a special address that always means “this machine”), which is where our QTXCore.js service resides that has it’s own HTTP/S and Websocket servers. The client (HTML5 part) is loaded in under a second from the core — and the experience is more or less identical to starting your ChromeBook or NAS box. Most modern NAS (network active storage) devices are much more than a file-server today. NAS boxes like those from Asustor Inc have HDMI out, ships with a remote control, and are designed to act as a media center. So you connect the NAS directly to your TV, and can watch movies and listen to music without any manual conversion etc.

In short, you can setup Quartex Media Desktop to do the exact same thing as ChromeOS does, booting straight into the web based desktop environment. The same desktop environment that is available over the network. So you are not limited to visiting your Cloud-Ripper machine via a browser from another computer; nor are you limited to just  using a dedicated machine. You can setup the system as you see fit.

Why should I assemble a Cloud-Ripper?

Getting a Cloud-Ripper is not forced on anyone. You can put together whatever spare hardware you have (or just run it locally under Windows). Since the services are extremely lightweight, any x86 PC will do. If you invest in a ODroid N2 board ($80 range) then you can install all the services on that if you like. So if you have no interest in clustering or building your own supercomputer, then any PC, Laptop or IOT single-board computer(s) will do. Provided it yields more or equal power as the XU4 (!)

What you will experience with a dedicated cluster, regardless of putting the boards in a nice cube, is that you get excellent performance for very little money. It is quite amazing what $200 can buy you in 2019. And when you daisy chain 5 ODroid XU4 boards together on a switch, those 5 cheap boards will deliver the same serving power as an x86 setup costing twice as much.

Jetson-Nano_3QTR-Front_Left_trimmed

The NVidia Jetson Nano SBC, one of the fastest boards available at under $100

Pico is offering 3 different packages. The most expensive option is the pre-assembled cube. This is for some reason priced at $750 which is completely absurd. If you can operate a screwdriver, then you can assemble the cube yourself in less than an hour. So the starter-kit case which costs $259 is more than enough.

Next, you can buy the XU4 boards directly from Hardkernel for $40 a piece, which will set you back $200. If you order the Pico 5H case as a kit, that brings the sub-total up to $459. But that price-tag includes everything you need except sd-cards. So the kit contains power-supply, the electrical wiring, a fast gigabit ethernet switch [built-into the cube], active cooling, network cables and power cables. You don’t need more than 8Gb sd-cards, which costs practically nothing these days.

Note: The Quartex Media Desktop “file-service” should have a dedicated disk. I bought a 256Gb SSD disk with a USB 3.0 interface, but you can just use a vanilla USB stick to store user-account data + user files.

As a bonus, such a setup is easy to recycle should you want to do something else later. Perhaps you want to learn more about Kubernetes? What about a docker-swarm? A freepascal build-server perhaps? Why not install FreeNas, Plex, and a good backup solution? You can set this up as you can afford. If 5 x ODroid XU4 is too much, then get 3 of them instead + the Pico 3H case.

So should Quartex Media Desktop not be for you, or you want to do something else entirely — then having 5 ODroid XU4 boards around the house is not a bad thing.

Oh and if you want some serious firepower, then order the Pico 5H kit for the NVidia Jetson Nano boards. Graphically those boards are beyond any other SoC on the market (in it’s price range). But as a consequence the Jetson Nano starts at $99. So for a full kit you will end up with $500 for the boards alone. But man those are the proverbial Ferrari of IOT.

Quartex Media Desktop, new compiler and general progress

September 11, 2019 3 comments

It’s been a few weeks since my last update on the project. The reason I dont blog that often about Quartex Media Desktop (QTXMD), is because the official user-group has grown to 2000+ members. So it’s easier for me to post developer updates directly to the audience rather than writing articles about it.

desktop_01

Quartex Media Desktop ~ a complete environment that runs on every device

If you haven’t bothered digging into the project, let me try to sum it up for you quickly.

Quick recap on Quartex Media Desktop

To understand what makes this project special, first consider the relationship between Microsoft Windows and a desktop program. The operating system, be it Windows, Linux or OSX – provides an infrastructure that makes complex applications possible. The operating-system offers functions and services that programs can rely on.

The most obvious being:

  • A filesystem and the ability to save and load data
  • A windowing toolkit so programs can be displayed and have a UI
  • A message system so programs can communicate with the OS
  • A service stack that takes care of background tasks
  • Authorization and identity management (security)

I have just described what the Quartex Media Desktop is all about. The goal is simple:

to provide for JavaScript what Windows and OS X provides for ordinary programs.

Just stop and think about this. Every “web application” you have ever seen, have all lacked these fundamental features. Sure you have libraries that gives you a windowing environment for Javascript, like Embarcadero Sencha; but im talking about something a bit more elaborate. Creating windows and buttons is easy, but what about ownership? A runtime environment has to keep track of the resources a program allocates, and make sure that security applies at every step.

Target audience and purpose

Take a second and think about how many services you use that have a web interface. In your house you probably have a router, and all routers can be administered via the browser. Sadly, most routers operate with a crude design and that leaves much to be desired.

router

Router interfaces for web are typically very limited and plain looking. Imagine what NetGear could do with Quartex Media Desktop instead

If you like to watch movies you probably have a Plex or Kodi system running somewhere in your house; perhaps you access that directly via your TV – or via a modern media system like Playstation 4 or XBox one. Both Plex and Kodi have web-based interfaces.

Netflix is now omnipresent and have practically become an institution in it’s own right. Netflix is often installed as an app – but the app is just a thin wrapper around a web-interface. That way they dont have to code apps for every possible device and OS out there.

If you commute via train in Scandinavia, chances are you buy tickets on a kiosk booth. Most of these booths run embedded software and the interface is again web based. That way they can update the whole interface without manually installing new software on each device.

plex-desktop-movies-1024x659

Plex is a much loved system. It is based on a mix of web and native technologies

These are just examples of web based interfaces you might know and use; devices that leverage web technology. As a developer, wouldn’t it be cool if there was a system that could be forked, adapted and provide advanced functionality out of the box?

Just imagine a cheap Jensen router with a Quartex Media Desktop interface! It could provide a proper UI interface with applications that run in a windowing environment. They could disable ordinary desktop functionality and run their single application in kiosk mode. Taking full advantage of the underlying functionality without loss of security.

And the same is true for you. If you have a great idea for a web based application, you can fork the system, adjust it to suit your needs – and deploy a cutting edge cloud system in days rather than months!

New compiler?

Up until recently I used Smart Mobile Studio. But since I have left that company, the matter became somewhat pressing. I mean, QTXMD is an open-source system and cant really rely on third-party intellectual property. Eventually I fired up Delphi, forked the latest  DWScript, and used that to roll a new command-line compiler.

desktop_02

Web technology has reached a level of performance that rivals native applications. You can pretty much retire Photoshop in favour of web based applications these days

But with a new compiler I also need a new RTL. Thankfully I have been coding away on the new RTL for over a year, but there is still a lot of work to do. I essentially have to implement the same functionality from scratch.

There will be more info on the new compiler / codegen when its production ready.

Progress

If I was to list all the work I have done since my last post, this article would be a small book. But to sum up the good stuff:

  • Authentication has been moved into it’s own service
  • The core (the main server) now delegates login messages to said service
  • We no longer rely on the Smart Pascal filesystem drivers, but use the raw node.js functions instead  (faster)
  • The desktop now use the Smart Theme engine. This means that we can style the desktop to whatever we like. The OS4 theme that was hardcoded will be moved into its own proper theme-file. This means the user can select between OS4, iOS, Android and Ubuntu styling. Creating your own theme-files is also possible. The Smart theme-engine will be replaced by a more elaborate system in QTX later
  • Ragnarok (the message api) messages now supports routing. If a routing structure is provided,  the core will relay the message to the process in question (providing security allows said routing for the user)
  • The desktop now checks for .info files when listing a directory. If a file is accompanied by an .info file, the icon is extracted and shown for that file
  • Most of the service layer now relies on the QTX RTL files. We still have some dependencies on the Smart Pascal RTL, but we are making good progress on QTX. Eventually  the whole system will have no dependencies outside QTX – and can thus be compiled without any financial obligations.
  • QTX has it’s own node.js classes, including server and client base-classes
  • Http(s) client and server classes are added to QTX
  • Websocket and WebSocket-Secure are added to QTX
  • TQTXHybridServer unifies http and websocket. Meaning that this server type can handle both orinary http requests – but also websocket connections on the same network socket. This is highly efficient for websocket based services
  • UDP classes for node.js are implemented, both client and server
  • Zero-Config classes are now added. This is used by the core for service discovery. Meaning that the child services hosted on another machine will automatically locate the core without knowing the IP. This is very important for machine clustering (optional, you can define a clear IP in the core preferences file)
  • Fixed a bug where the scrollbars would corrupt widget states
  • Added API functions for setting the scrollbars from hosted applications (so applications can tell the desktop that it needs scrollbar, and set the values)
  • .. and much, much more

I will keep you all posted about the progress — the core (the fundamental system) is set for release in december – so time is of the essence! Im allocating more or less all my free time to this, and it will be ready to rock around xmas.

When the core is out, I can focus solely on the applications. Everything from Notepad to Calculator needs to be there, and more importantly — the developer tools. The CloudForge IDE for developers is set for 2020. With that in place you can write applications for iOS, Android, Windows, OS X and Linux directly from Quartex Media Desktop. Nothing to install, you just need a modern browser and a QTX account.

The system is brilliant for small teams and companies. They can setup their own instance, communicate directly via the server (text chat and video chat is scheduled) and work on their products in concert.

Check out RemObjects Remoting SDK

July 22, 2019 3 comments

RemObjects Remoting SDK is one of those component packages that have become more than the sum of it’s part. Just like project Jedi has become standard equipment almost, Remoting SDK is a system that all Delphi and Freepascal developers should have in their toolbox.

ro_logo
In this article I’m going to present the SDK in broad strokes; from a viewpoint of someone who haven’t used the SDK before. There are still a large number of Delphi developers that don’t know it even exists – hopefully this post will shed some light on why the system is worth every penny and what it can do for you.

I should also add, that this is a personal blog. This is not an official RemObjects presentation, but a piece written by me based on my subjective experience and notions. We have a lot of running dialog at Delphi Developer on Facebook, so if I read overly harsh on a subject, that is my personal view as a Delphi Developer.

Stop re-inventing the wheel

Delphi has always been a great tool for writing system services. It has accumulated a vast ecosystem of non-visual components over the years, both commercial and non-commercial, and this allows developers to quickly aggregate and expose complex behavior — everything from graphics processing to databases, file processing to networking.

The challenge for Delphi is that writing large composite systems, where you have more than a single service doing work in concert, is not factored into the RTL or project type. Delphi provides a bare-bone project type for system services, and that’s it. Depending on how you look at it, it’s either a blessing or a curse. You essentially start on C level.

So fundamental things like IPC (inter process communication) is something you have to deal with yourself. If you want multi-tenancy that is likewise not supported out of the box. And all of this is before we venture into protocol standards, message formats and async vs synchronous execution.

The idea behind Remoting SDK is to get away from this style of low-level hacking. Without sounding negative, it provides the missing pieces that Delphi lacks, including the stuff that C# developers enjoy under .net (and then some). So if you are a Delphi developer who look over at C# with smudge of envy, then you are going to love Remoting SDK.

Say goodbye to boilerplate mistakes

Writing distributed servers and services is boring work. For each function you expose, you have to define the parameters and data-types in a portable way, then you have to implement the code that represents the exposed function and finally the interface itself that can be consumed by clients. The latter must be defined in a way that works with other languages too, not just Delphi. So while server tech in it’s essential form is quite simple, it’s the infrastructure that sets the stage of how quickly you can apply improvements and adapt to change.

For example, let’s say you have implemented a wonderful new service. It exposes 60 awesome functions that your customers can consume in their own work. The amount of boilerplate code for 60 distributed functions, especially if you operate with composite data types, is horrendous. It is a nightmare to manage and opens up for sloppy, unnecessary mistakes.

ide_int

After you install Remoting SDK, the service designer becomes a part of the IDE

This is where Remoting SDK truly shines. When you install the software, it integrates it’s editors and wizards closely with the Delphi IDE. It adds a ton of new project types, components and whatnot – but the most important feature is without a doubt the service designer.

bonjour

Start the service-designer in any server or service project and you can edit the methods, data types and interfaces your system expose to the world

As the name implies, the service designer allows you to visually define your services. Adding a new function is a simple click, the same goes for datatypes and structures (record types). These datatypes are exposed too and can be consumed from any modern language. So a service you make in Delphi can be used from C#, C/C++, Java, Oxygene, Swift (and visa-versa).

Auto generated code

A service designer is all good and well I hear you say, but what about that boilerplate code? Well Remoting SDK takes care of that too (kinda the point). Whenever you edit your services, the designer will auto-generate a new interface unit for you. This contains the classes and definitions that describe your service. It will also generate an implementation unit, with empty functions; you just need to fill in the blanks.

The designer is also smart enough not to remove code. So if you go in and change something, it won’t just delete the older implementation procedure. Only the params and names will be changed if you have already written some code.

bonjour_source

Having changed a service, hitting F9 re-generates the interface code automatically. Your only job is to fill in the code for each method in the implementation units. The SDK takes care of everything else for you

The service information, including the type information, is stored in a special file format called “rodl”. This format is very close to Microsoft WSDL format, but it holds more information. It’s important to underline that you can import the service directly from your servers (optional naturally) as WSDL. So if you want to consume a Remoting SDK service using Delphi’s ordinary RIO components, that is not a problem. Visual Studio likewise imports and consumes services – so Remoting SDK behaves identical regardless of platform or language used.

Remoting SDK is not just for Delphi, just to be clear on that. If you are presently using both Delphi and C# (which is a common situation), you can buy a license for both C# and Delphi and use whatever language you feel is best for a particular task or service. You can even get Remoting SDK for Javascript and call your service-stack directly from your website if you like. So there are a lot of options for leveraging the technology.

Transport is not content

OK so Remoting SDK makes it easy to define distributed services and servers. But what about communication? Are we boxed into RemObjects way of doing things?

The remoting framework comes with a ton of components, divided into 3 primary groups:

  • Servers
  • Channels (clients)
  • Messages

The reason for this distinction is simple: the ability to transport data, is never the same as the ability to describe data. For example, a message is always connected to a standard. It’s job is ultimately to serialize (represent) and de-serialize data according to a format. The server’s job is to receive a request and send a response. So these concepts are neatly decoupled for maximum agility.

As of writing the SDK offers the following message formats:

  • Binary
  • Post
  • SOAP
  • JSON

If you are exposing a service that will be consumed from JavaScript, throwing in a TROJSONMessage component is the way to go. If you expect messages to be posted from your website using ordinary web forms, then TROPostMessage is a perfect match. If you want XML then TROSOAPMessage rocks, and if you want fast, binary messages – well then there is TROBinaryMessage.

What you must understand is that you don’t have to pick just one! You can drop all 4 of these message formats and hook them up to your server or channel. The SDK is smart enough to recognize the format and use the correct component for serialization. So creating a distributed service that can be consumed from all major platforms is a matter of dropping components and setting a property.

channels

If you double-click on a server or channel, you can link message components with a simple click. No messy code snippets in sight.

Multi-tenancy out of the box

With the release of Rad-Server as a part of Delphi, people have started to ask what exactly multi-tenancy is and why it matters. I have to be honest and say that yes, it does matter if you are creating a service stack where you want to isolate the logic for each customer in compartments – but the idea that this is somehow new or unique is not the case. Remoting SDK have given users multi-tenancy support for 15+ years, which is also why I haven’t been too enthusiastic with Rad-Server.

Now don’t get me wrong, I don’t have an axe to grind with Rad-Server. The only reason I mention it is because people have asked how i feel about it. The tech itself is absolutely welcome, but it’s the licensing and throwing Interbase in there that rubs me the wrong way. If it could run on SQLite3 and was free with Enterprise I would have felt different about it.

mt-models

There are various models for multi-tenancy, but they revolve around the same principles

To get back on topic: multi-tenancy means that you can dynamically load services and expose them on demand. You can look at it as a form of plugin functionality. The idea in Rad-Server is that you can isolate a customer’s service in a separate package – and then load the package into your server whenever you need it.

ro_comps

Some of the components that ship with the system

The reason I dislike Rad-Server in this respect, is because they force you to compile with packages. So if you want to write a Rad-Server system, you have to compile your entire project as package-based, and ship a ton of .dpk files with your system. Packages is not wrong or bad per-se, but they open your system up on a fundamental level. There is nothing stopping a customer from rolling his own spoof package and potentially bypass your security.

There is also an issue with un-loading a package, where right now the package remains in memory. This means that hot-swapping packages without killing the server wont work.

Rad-Server is also hardcoded to use Interbase, which suddenly bring in licensing issues that rubs people the wrong way. Considering the price of Delphi in 2019, Rad-Server stands out as a bit of an oddity. And hardcoding a database into it, with the licensing issues that brings -just rendered the whole system mute for me. Why should I pay more to get less? Especially when I have been using multi-tenancy with RemObjects for some 15 years?

With Remoting SDK you have something called DLL servers, which does the exact same thing – but using ordinary DLL files (not packages!). You don’t have to compile your system with packages, and it takes just one line of code to make your main dispatcher aware of the loaded service.

This actually works so well that I use Remoting SDK as my primary “plugin” system. Even when I write ordinary desktop applications that has nothing to do with servers or services – I always try to compartmentalize features that could be replaced in the future.

For example, I’m a huge fan of ElevateDB, which is a native Delphi database engine that compiles directly into your executable. By isolating that inside a DLL as a service, my application is now engine agnostic – and I get a break from buying a truck load of components every time Delphi is updated.

Saving money

The thing about DLL services, is that you can save a lot of money. I’m actually using an ElevateDB license that was for Delphi 2007. I compiled the engine using D2007 into a DLL service — and then I consume that DLL from my more modern Delphi editions. I have no problem supporting or paying for components, that is right and fair, but having to buy new licenses for every single component each time Delphi is updated? This is unheard of in other languages, and I would rather ditch the platform all together than forking out $10k ever time I update.

dll_project

A DLL server can be used for many things if you are creative about it

While we are on the subject – Hydra is another great money saver. It allows you to use .net and Java libraries (both visual and non-visual) with Delphi. With Hydra you can design something in .net, compile it into a DLL file, and then use that from Delphi.

But — you can also compile things from Delphi, and use it in newer versions of Delphi. Im not forking out for a Developer Express update just to use what I have already paid for in the latest Delphi. I have one license, I compile the forms and components into a Hydra Module — and then use it from newer Delphi editions.

hydra

Hydra, which is a separate product, allows you to stuff visual components and forms inside a vanilla DLL. It allows cross  language use, so you can finally use Java and .net components inside your Delphi application

Bonjour support

Another feature I love is the zero configuration support. This is one of those things that you often forget, but that suddenly becomes important once you deploy a service stack on cluster level.

apple_bonjour_medium-e1485166557218Remoting SDK comes with support for Apple Bonjour, so if you want to use that functionality you have to install the Bonjour library from Apple. Once installed on your host machines, your RemObjects services can find each other.

ZeroConfig is not that hard to code manually. You can roll your own using UDP or vanilla messages. But getting service discovery right can be fiddly. One thing is broadcasting an UDP message saying “here I am”, it’s something else entirely to allow service discovery on cluster level.

If Bonjour is not your cup of tea, the SDK provides a second option, which is RemObjects own zero-config hub. You can dig into the documentation to find out more about this.

What about that IPC stuff you mentioned?

I mentioned IPC (inter process communication) at the beginning here, which is a must have if you are making a service stack where each member is expected to talk to the others. In a large server-system the services might not exist on the same, physical hardware either, so you want to take height for that.

With the SDK this is just another service. It takes 10 minutes to create a DLL server with the functionality to send and receive messages – and then you just load and plug that into all your services. Done. Finished.

Interestingly, Remoting SDK supports named-pipes. So if you are running on a Windows network it’s even easier. Personally I prefer to use a vanilla TCP/IP based server and channel, that way I can make use of my Linux blades too.

Building on the system

There is nothing stopping you from expanding the system that RemObjects have established. You are not forced to only use their server types, message types and class framework. You can mix and match as you see fit – and also inherit out your own variation if you need something special.

firm_foundation-720x340For example, WebSocket is an emerging standard that has become wildly popular. Remoting SDK does not support that out of the box, the reason is that the standard is practically identical to the RemObjects super-server, and partly because there must be room for third party vendors.

Andre Mussche took the time to implement a WebSocket server for Remoting SDK a few years back. Demonstrating in the process just how easy it is to build on the existing infrastructure. If you are already using Remoting SDK or want WebSocket support, head over to his github repository and grab the code there: https://github.com/andremussche/DelphiWebsockets

I could probably write a whole book covering this framework. For the past 15 years, RemObjects Remoting SDK is the first product I install after Delphi. It has become standard for me and remains an integral part of my toolkit. Other packages have come and gone, but this one remains.

Hopefully this post has tickled your interest in the product. No matter if you are maintaining a legacy service stack, or thinking about re implementing your existing system in something future-proof, this framework will make your life much, much easier. And it wont break the bank either.

You can visit the product page here: https://www.remotingsdk.com/ro/default.aspx

And you can check out the documentation here: https://docs.remotingsdk.com/

30% discount on all RemObjects products!

July 8, 2019 Leave a comment

This is brilliant. RemObjects is giving a whopping 30% discount on all products!

This means you can now pick up RemObjects Remoting Framework, Data Abstract, Hydra or the Elements compiler toolchain – with a massive 30% saving!

These are battle-hardened, enterprise level solutions that have been polished over years and they are in constant development. Each solution integrates seamlessly into Embarcadero Delphi and provides a smooth path to delivering quality products in days rather than weeks.

But you better hurry because it’s only valid for one week (!)

Use the coupon code: “DelphiDeveloper”

66825092_10156336639680906_8015817715019153408_o

Use the Delphi Developer coupon to get 30% discount – click here

 

Calling node.js from Delphi

July 6, 2019 1 comment

We got a good question about how to start a node.js program from Delphi on our Facebook group today (third one in a week?). When you have been coding for years you often forget that things like this might not be immediately obvious. Hopefully I can shed some light on the options in this post.

Node or chrome?

nodeJust to be clear: node.js has nothing to do with chrome or chromium embedded. Chrome is a web-browser, a completely visual environment and ecosystem.

Node.js is the complete opposite. It is purely a shell based environment, meaning that it’s designed to run services and servers, with emphasis on the latter.

The only thing node.js and chrome have in common, is that they both use the V8 JavaScript runtime engine to load, JIT compile and execute scripts at high speed. Beyond that, they are utterly alien to each other.

Can node.js be embedded into a Delphi program?

Technically there is nothing stopping a C/C++ developer from compiling the node.js core system as C++ builder compatible .obj files; files that can then be linked into a Delphi application through references. But this also requires a bit of scaffolding, like adding support for malloc_, free_ and a few other procedures – so that your .obj files uses the same memory manager as your Delphi code. But until someone does just that and publish it, im afraid you are stuck with two options:

  • Use a library called Toby, that keeps node.js in a single DLL file. This is the most practical way if you insist on hosting your own version of node.js
  • Add node.js as a prerequisite and give users the option to locate the node.exe in your application’s preferences. This is the way I would go, because you really don’t want to force users to stick with your potentially outdated or buggy build.

So yes, you can use toby and just add the toby dll file to your program folder, but I have to strongly advice against that. There is no point setting yourself up for maintaining a whole separate programming language, just because you want JavaScript support.

“How many in your company can write high quality WebAssembly modules?”

If all you want to do is support JavaScript in your application, then I would much rather install Besen into Delphi. Besen is a JavaScript runtime engine written in Freepascal. It is fully compatible with Delphi, and follows the ECMA standard to the letter. So it is extremely compatible, fast and easy to use.

Like all Delphi components Besen is compiled into your application, so you have no dependencies to worry about.

Starting a node.js script

The easiest way to start a node.js script, is to simply shell-execute out of your Delphi application. This can be done as easily as:

ShellExecute(Handle, 'open', PChar('node.exe'), pchar('script.js'), nil, SW_SHOW);

This is more than enough if you just want to start a service, server or do some work that doesn’t require that you capture the result.

If you need to capture the result, the data that your node.js program emits on stdout, there is a nice component in the Jedi Component Library. Also plenty of examples online on how to do that.

If you need even further communication, you need to look for a shell-execute that support pipes. All node.js programs have something called a message-channel in the Javascript world. In reality though, this is just a named pipe that is automatically created when your script starts (with the same moniker as the PID [process identifier]).

If you opt for the latter you have a direct, full duplex message channel directly into your node.js application. You just have to agree with yourself on a protocol so that your Delphi code understands what node.js is saying, and visa versa.

UDP or TCP

If you don’t want to get your hands dirty with named pipes and rolling your own protocol, you can just use UDP to let your Delphi application communicate with your node.js process. UDP is practically without cost since its fundamental to all networking stacks, and in your case you will be shipping messages purely between processes on localhost. Meaning: packets are never sent on the network, but rather delegated between processes on the same machine.

In that case, I suggest you ship in the port you want your UDP server to listen on, so that your node.js service acts as the server. A simple command-line statement like:

node.exe myservice.js 8090

Inside node.js you can setup an UDP server with very little fuzz:


function setupServer(port) {
  var os = require("os");
  var dgram = require("dgram");
  var socket = dgram.createSocket("udp4");

  var MULTICAST_HOST = "224.0.0.236";
  var BROADCAST_HOST = "255.255.255.255";
  var ALL_PORT = 60540;
  var MULTICAST_TTL = 1; // Local network

  socket.bind(port);
  socket.on('listening', function() {
    socket.setMulticastLoopback(true);
    socket.setMulticastTTL(MULTICAST_TTL);
    socket.addMembership(multicastHost);
    if(broadcast) { socket.setBroadcast(true); }
  });
  socket.on('message', parseMessage);
}

function parseMessage(message, rinfo) {
try {
  var messageObject = JSON.parse(message);
  var eventType = messageObject.eventType;
  } catch(e) {
  }
}

Note: the code above assumes a JSON text message.

You can then use any Delphi UDP client to communicate with your node.js server, Indy is good, Synapse is a good library with less overhead – there are many options here.

Do I have to learn Javascript to use node.js?

If you download DWScript you can hook-up the JS-codegen library (see library folder in the DWScript repository), and use that to compile DWScript (object pascal) to kick-ass Javascript. This is the same compiler that was used in Smart Mobile Studio.

“Adding WebAssembly to your resume is going to be a hell of a lot more valuable in the years to come than C# or Java”

Another alternative is to use Freepascal, they have a pas2js project where you can compile ordinary object-pascal to javascript. Naturally there are a few things to keep in mind, both for DWScript and Freepascal – like avoiding pointers. But clean object pascal compiles just fine.

If JavaScript is not your cup of tea, or you simply don’t have time to learn the delicate nuances between the DOM (document object model, used by browsers) and the 100% package oriented approach deployed by node.js — then you can just straight up to webassembly.

RemObjects Software has a kick-ass webassembly compiler, perfect if you dont have the energy or time to learn JavaScript. As of writing this is the fastest and most powerful toolchain available. And I have tested them all.

WebAssembly, no Javascript needed

RO-Single-Gear-512You might remember Oxygene? It used to be shipped with Delphi as a way to target Microsoft CLR (common language runtime) and the .net framework.

Since then Oxygene and the RemObjects toolchain has evolved dramatically and is now capable of a lot more than CLR support.

  • You can compile to raw, llvm optimized machine code for 8 platforms
  • You can compile to CLR/.Net
  • You can compile to Java bytecodes
  • You can compile to WebAssembly!

WebAssembly is not Javascript, it’s important to underline that. WebAssembly was created especially for developers using traditional languages, so that traditional compilers can emit web friendly, binary code. Unlike Javascript, WebAssembly is a purely binary format. Just like Delphi generates machine-code that is linked into a final executable, WebAssembly is likewise compiled, linked and emitted in binary form.

If that sounds like a sales pitch, it’s not. It’s a matter of practicality.

  • WebAssembly is completely barren out of the box. The runtime environment, be it V8 for the browser or V8 for node.js, gives you nothing out of the box. You don’t even have WriteLn() to emit text.
  • Google expects compiler makers to provide their own RTL functions, from the fundamental to the advanced. The only thing V8 gives you, is a barebone way of referencing objects and functions on the other side, meaning the JS and DOM world. And that’s it.

So the reason i’m talking a lot about Oxygene and RemObjects Elements (Elements is the name of the compiler toolchain RemObjects offers), is because it ships with an RTL. So you are not forced to start on actual, literal assembly level.

studio

If you don’t want to study JavaScript, Oxygene and Elements from RemObjects is the solution

RemObjects also delivers a DelphiVCL compatibility framework. This is a clone of the Delphi VCL / Freepascal LCL. Since WebAssembly is still brand new, work is being done on this framework on a daily basis, with updates being issued all the time.

Note: The Delphi VCL framework is not just for WebAssembly. It represents a unified framework that can work anywhere. So if you switch from WebAssembly to say Android, you get the same result.

The most important part of the above, is actually not the visual stuff. I mean, having HTML5 visual controls is cool – but chances are you want to use a library like Sencha, SwiftUI or jQueryUI to compose your forms right? Which means you just want to interface with the widgets in the DOM to set and get values.

jQuery UI Bootstrap

You probably want to use a fancy UI library, like jQuery UI. This works perfectly with Elements because you can reference the controls from your WebAssembly module. You dont have to create TButton, TListbox etc manually

The more interesting stuff is actually the non-visual code you get access to. Hundreds of familiar classes from the VCL, painstakingly re-created, and usable from any of the 5 languages Elements supports.

You can check it out here: https://github.com/remobjects/DelphiRTL

Skipping JavaScript all together

I dont believe in single languages. Not any more. There was a time when all you needed was Delphi and a diploma and you were set to conquer the world. But those days are long gone, and a programmer needs to be flexible and have a well stocked toolbox.

At least try the alternatives before you settle on a phone

Knowing where you want to be is half the journey

The world really don’t need yet-another-c# developer. There are millions of C# developers in India alone. C# is just “so what?”. Which is also why C# jobs pays less than Delphi or node.js system service jobs.

What you want, is to learn the things others avoid. If JavaScript looks alien and you feel uneasy about the whole thing – that means you are growing as a developer. All new things are learned by venturing outside your comfort zone.

How many in your company can write high quality WebAssembly modules?

How many within one hour driving distance from your office or home are experts at WebAssembly? How many are capable of writing industrial scale, production ready system services for node.js that can scale from a single instance to 1000 instances in a large, clustered cloud environment?

Any idiot can pick up node.js and knock out a service, but with your background from Delphi or C++ builder you have a massive advantage. All those places that can throw an exception that JS devs usually ignore? As a Delphi or Oxygene developer you know better. And when you re-apply that experience under a different language, suddenly you can do stuff others cant. Which makes your skills valuable.

qtx

The Quartex Media Desktop have made even experienced node / web developers gasp. They are not used to writing custom-controls and large-scale systems, which is my advantage

So would you learn JavaScript or just skip to WebAssembly? Honestly? Learn a bit of both. You don’t have to be an expert in JavaScript to compliment WebAssembly. Just get a cheap book, like “Node.js for beginners” and “JavaScript the good parts” ($20 a piece) and that should be more than enough to cover the JS side of things.

Adding WebAssembly to your resume and having the material to prove you know your stuff, is going to be a hell of a lot more valuable in the years to come than C#, Java or Python. THAT I can guarantee you.

And, we have a wicked cool group on Facebook you can join too: Click here to visit RemObjects Developer.

 

Getting into Node.js from Delphi

July 1, 2019 1 comment

Delphi is one of the best development toolchains for Windows. I have been an avid fan of Delphi since it was first released, and before that – Turbo Pascal too. Delphi has a healthy following – and despite popular belief, Delphi scores quite well on the Tiobe Index.

As cool and efficient as Delphi might be, there are situations where native code wont work. Or at the very least, be less efficient than the alternatives. Delphi has a broad wingspan, from low-level assembler all the way to classes and generics. But JavaScript and emerging web technology is based on a completely different philosophy, one where native code is regarded as negative since it binds you to hardware.

Getting to grips with the whole JavaScript phenomenon, be it for mobile, embedded or back-end services, can be daunting if all you know is native code. But thankfully there are alternatives that can help you become productive quickly, something I will brush over in this post.

JavaScript without JavaScript

Before we dig into the tools of the trade, I want to cover alternative ways of enjoying the power of node.js and Javascript. Namely by using compilers that can convert code from a traditional language – and emit fully working JavaScript. There are a lot more options than you think:

qtx

Quartex Media Desktop is a complete environment written purely in JavaScript. Both Server, Cluster and front-end is pure JavaScript. A good example of what can be done.

  • Swift compiles for JavaScript, and Apple is doing some amazing things with the new and sexy SwiftUI tookit. If you know your way around Swift, you can compile for Javascript
  • Go can likewise be compiled to JS:
    • RemObjects Elements supports the Go language. Elements can target both native (llvm), .Net, Java and WebAssembly.
    • Go2Js
    • GopherJs
    • TARDISgo
  • C/C++ can be compiled to asm.js courtesy of EmScripten. It uses clang to first compile your code to llvm bitcode, and then it converts that into asm.js. You have probably seen games like Quake run in the browser? That was asm.js, a kind of precursor to WebAssembly.
  • NS Basic compiles for JavaScript, this is a Visual Basic 6 style environment with its own IDE even

For those coming straight from Delphi, there are a couple of options to pick from:

  • Freepascal (pas2js project)
  • DWScript compiles code to JavaScript, this is the same compiler that we used in Smart Pascal earlier
  • Oxygene, the next generation object-pascal from RemObjects compiles to WebAssembly. This is by far the best option of them all.

studio

I strongly urge you to have a look at Elements, here running in Visual Studio

JavaScript, Asm.js or WebAssembly?

Asm.js is by far the most misunderstood technology in the JavaScript ecosystem, so let me just cover that before we move on:

A few years back JavaScript gained support for memory buffers and typed arrays. This might not sound very exciting, but in terms of speed – the difference is tremendous. The default variable type in JavaScript is what Delphi developers know as Variant. It assumes the datatype of the values you assign to it. Needless to say, there is a lot of overhead when working with variants – so JavaScript suddenly getting proper typed arrays was a huge deal.

It was then discovered that JavaScript could manipulate these arrays and buffers at high speed, providing it only used a subset of the language. A subset that the JavaScript runtime could JIT compile more easily (turn into machine-code).

So what the EmScripten team did was to implement a bytecode based virtual-machine in Javascript, and then they compile C/C++ to bytecodes. I know, it’s a huge project, but the results speak for themselves — before WebAssembly, this was as fast as it got with JavaScript.

WebAssembly

WebAssembly is different from both vanilla JavaScript and Asm.js. First of all, it’s executed at high speed by the browser itself. Not like asm.js where these bytecodes were executed by JavaScript code.

water

Water is a fast, slick and platform independent IDE for Elements. The same IDE for OS X is called Fire. You can use RemObjects Elements from either Visual Studio or Water

Secondly, WebAssembly is completely JIT compiled by the browser or node.js when loading. It’s not like Asm.js where some parts are compiled, others are interpreted. WebAssembly runs at full speed and have nothing to do with traditional JavaScript. It’s actually a completely separate engine.

Out of all the options on the table, WebAssembly is the technology with the best performance.

Kits and strategies

The first thing you need to be clear about, is what you want to work with. The needs and requirements of a game developer will be very different from a system service developer.

Here are a couple of kits to think about:

  • Mobile developer
    • Implement your mobile applications using Oxygene, compiling for WebAssembly (Elements)
    • RemObjects Remoting SDK for client / server communication
    • Use Freepascal for vanilla JavaScript scaffolding when needed
  • Service developer
    • Implement libraries in Oxygene to benefit from the speed of WebAssembly
    • Use RemObjects Data Abstract to make data-access uniform and fast
    • Use Freepascal for boilerplate node.js logic
  • Desktop developer
    • For platform independent desktop applications, WebAssembly is the way to go. You will need some scaffolding (plain Javascript) to communicate with the application host  – but the 99.9% of your code will be better under WebAssembly.
    • Use Cordova / Phonegap to “bundle” your WebAssembly, HTML5 files and CSS styling into a single, final executable.

The most important part to think about when getting into JavaScript, is to look closely at the benefits and limitation of each technology.

WebAssembly is fast, wicked fast, and let’s you write code like you are used to from Delphi. Things like pointers etc are supported in Elements, which means ordinary code that use pointers will port over with ease. You are also not bound on hand-and-feet to a particular framework.

For example, EmScripten for C/C++ have almost nothing in terms of UI functionality. The visual part is a custom build of SDL (simple directmedia layer), which fakes the graphics onto an ordinary HTML5 canvas. This makes EmScripten a good candidate for porting games written in C/C++ to the web — but it’s less than optimal for writing serious applications.

Setting up the common tools

So far we have looked at a couple of alternatives for getting into the wonderful world of JavaScript in lieu of other languages. But what if you just want to get started with the typical tools JS developers use?

vscode

Visual Studio Code is a pretty amazing code-editor

The first “must have” is Visual Studio Code. This is actually a great example of what you can achieve with JavaScript, because the entire editor and program is written in JavaScript. But I want to stress that this editor is THE editor to get. The way you work with files in JS is very different from Delphi, C# and Java. JavaScript projects are often more fragmented, with less code in each file – organized by name.

typescript

TypeScript was invented by Anders Hejlsberg, who also made Delphi and C#

The next “must have” is without a doubt TypeScript. Personally im not too fond of TypeScript, but if ordinary JavaScript makes your head hurt and you want classes and ordinary inheritance, then TypeScript is a step up.

assemblyscriptNext on the list is AssemblyScript. This is a post-processor for TypeScript that converts your code into WebAssembly. It lacks much of the charm and elegance of Oxygene, but I suspect that has to do with old habits. When you have been reading object-pascal for 20 years, you feel more at home there.

nodeYou will also need to install node.js, which is the runtime engine for running JavaScript as services. Node.js is heavily optimized for writing server software, but it’s actually a brilliant way to write services that are multi-platform. Because Node.js delivers the same behavior regardless of underlying operating system.

phonegapAnd finally, since you definitely want to convert your JavaScript and/or WebAssembly into a stand-alone executable: you will need Adobe Phonegap.

Visual Studio

No matter if you want to enter JavaScript via Elements or something else, Visual Studio will save you a lot of time, especially if you plan on targeting Azure or Amazon services. Downloading and installing the community edition is a good idea, and you can use that while exploring your options.

dotnet-visual-studio

When it comes to writing system services, you also want to check out NPM, the node.js package manager. The JavaScript ecosystem is heavily package oriented – and npm gives you some 800.000 packages to play with free of charge.

Just to be clear, npm is a shell command you use to install or remove packages. NPM is also a online repository of said packages, where you can search and find what you need. Most packages are hosted on github, but when you install a package locally into your application folder – npm figures out dependencies etc. automatically for you.

Books, glorious books

41QSvp9fTcL._SX331_BO1,204,203,200_Last but not least, get some good books. Seriously, it will save you so much time and frustration. Amazon have tons of great books, be it vanilla JavaScript, TypeScript, Node.js — pick some good ones and take the time to consume the material.

And again, I strongly urge you to have a look at Elements when it comes to WebAssembly. WebAssembly is a harsh and barren canvas, and being able to use the Elements RTL is a huge boost.

But regardless of path you pick, you will always benefit from learning vanilla JavaScript.

 

Two new groups in the Developer family

July 1, 2019 2 comments

Delphi Developer is a group on Facebook that have been going strong for 12+ years. It was one of the first groups on Facebook, created the same week that Facebook allowed groups. With that group well established, it’s time to expand and clean up the feed.

RO-Single-Gear-512Last month I introduced a new group, RemObjects Developer, which is a group for developers that use RemObjects components, like the Remoting SDK, Data Abstract and/or Hydra – but more in particular, developers using Oxygene, C#, Swift, Java or Go via Elements (RemObjects compiler toolchain).

Two new groups

To further simplify syndication, and clean up the feeds (which so far has been a pot-purrey of many topics, dialects and products) an additional two groups is now in place:

Obviously there will be some overlapping. Since FPC and Delphi has much in common and are for the most part compatible, some news will be shared between those groups. But all in all this is to clean up the newsfeed which has so far been a mix and match of everything.

org

Simple overview of the groups

Node.js Developer is not meant to be purely about vanilla JavaScript. Node.js is ultimately a JavaScript runtime-engine. Which means you can use it to run or host WebAssembly libraries (as produced by Oxygene), or generate code via DWScript or Freepascal. You can think of it as a service-host if you like.

So if you are writing WebAssembly applications using Elements, then the node.js group will no doubt be interesting too. Same goes for DWScript users, Smart Pascal users and Freepascal users – providing web tech is what they like.

What is this Quartex Components?

It’s easier to manage multiple groups if you attach them to a parent-page. So if you wonder why all the groups says “by Quartex Components”, that is just a top-level page that helps me deal with with syndication. For some reason Facebook’s API only works for pages, not groups. So it’s impossible to auto-import news (for example) without a page.

The name, “Quartex Components” is ultimately the name of my personal company. I used to produce security components for Delphi, but decided to open-source those for the community.

So Quartex Components is just an organizational element.

RemObjects VCL, mind blown!

June 12, 2019 12 comments

For a guy that spends most of his time online, and can talk for hours about the most nerdy topics known to mankind – being gobsmacked and silenced is a rare event. But this morning that was exactly what happened.

Now, Marc Hoffman has blogged regularly over the years regarding the evolution of the RemObjects toolchain; explaining how they decoupled the parts that make up a programming language, such as syntax, rtl and target, but I must admit haven’t really digested the full implications of that work.

Like most developers I have kept my eyes on the parts relevant for me, like the Remoting SDK, Data Abstract and Javascript support. Before I worked at Embarcadero I pretty much spent 10 years contracting -and building Smart Mobile Studio on the side together with the team at The Smart Company Inc.

xo

Smart Pascal gained support for RemObjects SDK servers quite early

Since both the Remoting SDK and Data Abstract were part of our toolbox as Delphi developers, those were naturally more immediate than anything else. We also added support for RemObjects Remoting SDK inside Smart Mobile Studio, so that people could call existing services from their Javascript applications.

Oxygene then

Like most Delphi developers I remember testing Oxygene Pascal when I bought Delphi 2005. Back then Oxygene was licensed by Borland under the “Prism” name and represented their take on dot net support. I was very excited when it came out, but since my knowledge of the dot net framework was nil, I was 100% relient on the documentation.

In many ways Oxygene was a victim of Rad Studio’s abhorrent help-file system. Documentation for Rad Studio (especially Delphi) up to that point had been exemplary since Delphi 4; but by the time Rad Studio 2005 came out, the bloat had reached epic levels. Even for me as a die-hard Delphi fanatic, Delphi 2005 and 2006 was a tragic experience.

image

Removing Oxygene was a monumental mistake

I mean, when it takes 15 minutes (literally) just to open the docs, then learning a whole new programming paradigm under those conditions was quite frankly impossible. Like most Delphi developers I was used to Delphi 7 style documentation, where the docs were not just reference material – but actually teaches you the language itself.

In the end Oxygene remained very interesting, but with a full time job, deadlines and kids to take care of, I stuck to what I knew – namely the VCL.

Oxygene today

Just like Delphi has evolved and improved radically since 2005, Oxygene has likewise evolved above and beyond its initial form. Truth be told, we copied a lot of material from Oxygene when we made Smart Pascal, so I feel strangely at home with Oxygene even after a couple of days. The documentation for Oxygene Pascal (and Elements as a whole) is very good: https://docs.elementscompiler.com/Oxygene/

But Oxygene Pascal, while the obvious “first stop” for Delphi developers looking to expand their market impact, is more than “just a language”. It’s a language that is a part of a growing family of languages that RemObjects support and evolve.

As of writing RemObjects offers the following languages. So even if you don’t have a background in Delphi, or perhaps migrated from Delphi to C# years ago – RemObjects will have solutions and benefits to offer:

  • Oxygene (object pascal)
  • C#
  • Swift
  • Java

water

Water is a sexy, slim new IDE for RemObjects languages on Windows. For the OS X version you want to download Fire.

And here is the cool thing: when you hear “Java” you automatically expect that you are bound hands and feet to the Java runtime-libraries right? Same also with C#, you expect C# to be purely limited to the dot-net framework. And if you like me dabbed in Oxygene back in 2005-2006, you probably think Oxygene is purely a dot-net adapted version of Object Pascal right? But RemObjects have turned that on it’s head!

Remember the decoupling I mentioned at the beginning of this post? What that means in practical terms is that they have separated each language into three distinct parts:

  1. The syntax
  2. The RTL
  3. The target

What this means, is that you can pick your own combinations!

Let’s say you are coming from Delphi. You have 20 years of Object Pascal experience under your belt, and while you dont mind learning new things – Object Pascal is where you will be most productive.

Well in that case picking Oxygene Pascal covers the syntax part. But you don’t have to use the dot-net framework if you don’t want to. You can mix and match these 3 parts as you see fit! Let’s look at some combinations you could pick:

  • Oxygene Pascal -> dot net framework -> CIL
  • Oxygene Pascal -> “VCL” -> CIL
  • Oxygene Pascal -> “VCL” -> WinAPI
  • Oxygene Pascal -> “VCL” -> WebAssembly

(*) The “VCL” here is a compatibility RTL closely modeled on the Freepascal LCL and Delphi VCL. This is written from scratch and contains no proprietary code. It is purely to get people productive faster.

The whole point of this tripartite decoupling is to allow developers to maximize the value of their existing skill-set. If you know Object Pascal then that is a natural starting point for you. If you know the VCL then obviously the VCL compatibility RTL is going to help you become productive much faster than calling WinAPI on C level. But you can, if you like, go all native. And you can likewise ignore native and opt for WebAssembly.

Sound cool? Indeed it is! But it gets better, let’s look at some of the targets:

  • Microsoft Windows
  • Apple OS X
  • Apple iOS
  • Apple WatchOS
  • Android
  • Android wearables
  • Linux x86 / 64
  • Linux ARM
  • tvOS
  • WebAssembly
  • * dot-net
  • * Java

In short: Pick the language you want, pick the RTL or framework you want, pick the target you want — and start coding!

(*) dot-net and Java are not just frameworks, they are also targets since they are Virtual Machines. WebAssembly also fall under the VM category, although the virtual machine there is bolted into Chrome and Firefox (also node.js).

Some example code

Webassembly is something that interest me more than native these days. Sure I love the speed that native has to offer, but since Javascript has become “the defacto universal platform”, and since most of my work privately is done in Javascript – it seems like the obvious place to start.

Webassembly is a bit like Javascript was 10 years ago. I remember it was a bit of a shock coming from Delphi. We had just created Smart Mobile Studio, and suddenly we realized that the classes and object the browser had to offer were close to barren. We were used to the VCL after all. So my work there was basically to implement something with enough similarity to the VCL to be familiar to to Delphi developer, without wandering too far away from established JS standards.

Webassembly is roughly in the same ballpark. Webassembly is just a runtime engine. It doesn’t give you all those nice and helpful classes out of the box. You are expected to either write that yourself – or (as luck would have it) rely on what language vendors provide.

RemObjects have a lot to offer here, because their “Delphi VCL” compatibility RTL compiles just fine for Webassembly. There is no form designer though, but I haven’t used a form designer in years. I prefer to do everything in code because that’s ultimately what works when your codebase grows large enough anyways. Even my Delphi projects are done mainly as raw code, because I like to have the option to compile with Freepascal and Lazarus.

My first test code for Oxygene Pascal with Webassembly as the target is thus very bare-bone. If there is something that has bugged me to no end, it’s that bloody HTML5 canvas. It’s a powerful thing, but it’s also overkill for per-pixel operations. So I figured that a nice, ad-hoc DIB (device independent bitmap) class will do wonders.

Note: Oxygene supports pointers, even under WebAssembly (!), but out of old habit I have avoided it. I want my code to compile for all the targets, without marking a class as “unsafe” in the dot-net paradigm. So I have avoided pointers and just use offsets instead.

namespace qtxlib;

interface

type

  // in-memory pixel format
  TPixelFormat = public (
      pf8bit  = 0,  //___8 -- palette indexed
      pf15bit = 1,  //_555 -- 15 bit encoded
      pf16bit = 2,  //_565 -- 16 bit encoded
      pf24bit = 3,  //_888 -- 24 bit native
      pf32bit = 4   //888A -- 32 bit native
      );

  TPixelBuffer = public class
  private
    FPixels:  array of Byte;
    FDepthLUT: array of Integer;
    FScanLUT: array of Integer;
    FStride:  Integer;
    FWidth:   Integer;
    FHeight:  Integer;
    FBytes:   Integer;
    FFormat:  TPixelFormat;
  protected
    function  CalcStride(const Value, PixelByteSize, AlignSize: Integer): Integer;
    function  GetEmpty: Boolean;
  public
    property  Width: Integer read FWidth;
    property  Height: Integer read FHeight;
    property  Stride: Integer read FStride;
    property  &Empty: Boolean read GetEmpty;
    property  BufferSize: Integer read FBytes;
    property  PixelFormat: TPixelFormat read FFormat;
    property  Buffer[const index: Integer]: Byte read (FPixels[&index]) write (FPixels[&index]);

    function  OffsetForPixel(const dx, dy: Integer): Integer;
    procedure Alloc(NewWidth, NewHeight: Integer; const PxFormat: TPixelFormat);
    procedure Release();

    function Read(Offset: Integer; ByteLength: Integer): array of Byte;
    procedure Write(Offset: Integer; const Data: array of Byte);

    constructor Create; virtual;

    finalizer;
    begin
      if not GetEmpty() then
        Release();
    end;
end;

TColorMixer = public class
end;

TPainter = public class
private
  FBuffer:    TPixelBuffer;
public
  property    PixelBuffer: TPixelBuffer read FBuffer;

  constructor Create(const PxBuffer: TPixelBuffer); virtual;
end;

implementation

//##################################################################################
// TPainter
//##################################################################################

constructor TPainter.Create(const PxBuffer: TPixelBuffer);
begin
  inherited Create();
  if PxBuffer  nil then
    FBuffer := PxBuffer
  else
    raise new Exception("Pixelbuffer cannot be NIL error");
end;

//##################################################################################
// TPixelBuffer
//##################################################################################

constructor TPixelBuffer.Create;
begin
  inherited Create();
  FDepthLUT := [1, 2, 2, 3, 4];
end;

function TPixelBuffer.GetEmpty: Boolean;
begin
  result := length(FPixels) = 0;
end;

function TPixelBuffer.OffsetForPixel(const dx, dy: integer): Integer;
begin
  if length(FPixels) > 0 then
  begin
    result := dy * FStride;
    inc(result, dx * FDepthLUT[FFormat]);
  end;
end;

procedure TPixelBuffer.Write(Offset: Integer; const Data: array of Byte);
begin
  for each el in Data do
  begin
    FPixels[Offset] := el;
    inc(Offset);
  end;
end;

function TPixelBuffer.Read(Offset: Integer; ByteLength: Integer): array of Byte;
begin
  result := new Byte[ByteLength];
  var xOff := 0;
  while ByteLength > 0 do
  begin
    result[xOff] := FPixels[Offset];
    dec(ByteLength);
    inc(Offset);
    inc(xOff);
  end;
end;

procedure TPixelBuffer.Alloc(NewWidth, NewHeight: Integer; const PxFormat: TPixelFormat);
begin
  if not GetEmpty() then
    Release();

  if NewWidth < 1 then
    raise new Exception("Invalid width error");

  if NewHeight  0 then
    result := ( (Result + AlignSize) - xFetch );
end;

end.

This code is just meant to give you a feel for the dialect. I have used a lot of “Delphi style” coding here, so chances are you will hardly see any difference bar namespaces and a funny looking property declaration.

Stay tuned for more posts as I explore the different aspects of Oxygene and webassembly in the days to come 🙂

RemObjects Remoting SDK?

June 3, 2019 Leave a comment

Reading this you could be forgiven for thinking that I must promote RemObjects products, It’s my job now right? Well yes, but also no.

dataabstract-illustration-rework-ro-1100The thing is, I’m really not “traveling salesman” material by any stretch of the imagination. My tolerance for bullshit is ridiculously low, and being practical of nature I loath fancy products that cost a fortune yet deliver nothing but superficial fluff.

The reasons I went to work at RemObjects are many, but most of all it’s because I have been an avid supporter of their products since they launched. I have used and seen their products in action under intense pressure, and I have come to put some faith in their solutions.

Trying to describe what it’s like to write servers that should handle thousands of active user “with or without” RemObjects Remoting SDK is exhausting, because you end up sounding like a fanatic. Having said that, I feel comfortable talking about the products because I speak from experience.

I will try to outline some of the benefits here, but you really should check it out yourself. You can download a trial directly here: https://www.remotingsdk.com/ro/

Remoting framework, what’s that?

RemObjects Remoting framework (or “RemObjects SDK” as it was called earlier) is a framework for writing large-scale RPC (remote procedure call) servers and services. Unlike the typical solutions available for Delphi and C++ builder, including those from Embarcadero I might add, RemObjects framework stands out because it distinguishes between transport, host and message-format – and above all, it’s sheer quality and ease of use.

compo

RemObjects Remoting SDK ships with a rich selection of channels and message formats

This separation between transport, host and message-format makes a lot of sense, because the parameters and data involved in calling a server-method, shouldn’t really be affected by how it got there.

And this is where the fun begins because the framework offers you a great deal of different server types (channels) and you can put together some interesting combinations by just dragging and dropping components.

How about JSON over email? Or XML over pipes?

The whole idea here is that you don’t have to just work with one standard (and pay through the nose for the privilege). You can mix and match from a rich palette of transport mediums and message-formats and instead focus on your job; to deliver a kick-ass product.

And should you need something special that isn’t covered by the existing components, inheriting out your own channel or message classes is likewise a breeze. For example, Andre Mussche have some additional components on GitHub that adds a WebSocket server and client. So there is a lot of room for expanding and building on the foundation provided by RemObjects.

And this is where RemObjects has the biggest edge (imho), namely that their solutions shaves weeks if not months off your development time. And the central aspect of that is their integrated service designer.

Integration into the Delphi IDE

Dropping components on a form is all good and well, but the moment you start coding services that deploy complex data-types (records or structures) the amount of boilerplate code can become overwhelming.

The whole point of a remoting framework is that it should expose your services to the world. Someone working in .net or Java on the other side of the planet should be able to connect, consume and invoke your services. And for that to happen every minute detail of your service has to follow standards.

61855462_10156255129755906_1396051777802993664_o

The RemObjects Service Builder integrates directly into the Delphi IDE

When you install RemObjects SDK, it also integrates into the Delphi IDE. And one of the features it integrates is a complete, separate service designer. The designer can also be used outside of the Delphi IDE, but I cannot underline enough how handy it is to be able to design your services visually, right there and then, in the Delphi IDE.

This designer doesn’t just help you design your service description (RemObjects has their own RODL file-format, which is a bit like a Microsoft WSDL file), the core purpose is to auto-generate all the boilerplate code for you — directly into your Delphi project (!)

So instead of you having to spend a week typing boilerplate code for your killer solution, you get to focus on implementing the actual methods (which is what you are supposed to be doing in the first place).

DLL services, code re-use and multi-tenancy

The idea of multi-tenancy is an interesting one. One that I talked about with regards to Rad-Server both in Oslo and London before christmas. But Rad-Server is not the only system that allows for multi-tenancy. I was doing multi-tenancy with RemObjects SDK some 14 years ago (if not earlier).

Remember how I said the framework distinguishes between transport, message and host? That last bit, namely host, is going to change how you write applications.

When you install the framework, it registers a series of custom project types inside the Delphi IDE. So if you want to create a brand new RemObjects SDK server project, you can just do that via the ordinary File->New->Other menu option.

One of the project types is called a DLL Server. Which literally means you get to isolate a whole service library inside a single DLL file! You can then load in this DLL file and call the functions from other projects. And that is, ultimately, the fundamental principle for multi-tenancy.

And no, you don’t have to compile your project with external packages for this to work. The term “dll-server” can also be a bit confusing, because we are not compiling a network server into a DLL file, we are placing the code for a service into a DLL file. I used this project type to isolate common code, so I wouldn’t have to copy unit-files all over the place when delivering the same functionality.

It’s also a great way to save money. Don’t want to pay for that new upgrade? Happy with the database components you have? Isolate them in a DLL-Server and continue to use the code from your new Delphi edition. I have Delphi XE3 Database components running inside a RemObjects DLL-Server that I use from Delphi XE 10.3.

project_types

DLL server is awesome and elegantly solves real-life problems out of the box

In my example I was doing business-logic for our biggest customers. Each of them used the same database, but they way they registered data was different. The company I worked for had bought up these projects (and thus their customers with them), and in order to keep the customers happy we couldn’t force them to re-code their systems to match ours. So we had to come up with a way to upgrade our technology without forcing a change on them.

The first thing I did was to create a “DLL server” that dealt with the database. It exposed methods like openTable(), createInvoice(), getInvoiceById() and so on. All the functions I would need to work with the data without getting my fingers dirty with SQL outside the DLL. So all the nitty gritty of SQL components, queries and whatnot was neatly isolated in that DLL file.

I then created separate DLL-Server projects for each customer, implemented their service interfaces identical to their older API. These DLL’s directly referenced the database library for authentication and doing the actual work.

62174838_10156255134895906_9195165500563259392_n

When integrated with the IDE, you are greeted with a nice welcome window when you start Delphi. Here you can open examples or check out the documentation

Finally, I wrapped it all up in a traditional Windows system service, which contained two different server-channels and the message formats they needed. When the service was started it would simply load in the DLL’s and manually register their services and types with the central channel — and voila, it worked like a charm!

Rock solid

Some 10 years after I delivered the RemObjects based solution outlined above, I got a call from my old employer. They had been victim of a devastating cyber attack. I got a bit anxious as he went on and on about damages and costs, fearing that I had somehow contributed to the situation.

targets

But it turned out he called to congratulate me! Out of all the services in their server-park, mine were the only ones left standing when the dust settled.

The RemObjects payload balancer had correctly dealt with both DDOS and brute force attacks, and the hackers were left wanting at the gates.

New job, new office, new adventures

May 12, 2019 5 comments

It’s been roughly 4 weeks since I posted a status report on Amibian.js. I normally keep people up-to-date on facebook (the “Amiga Disrupt” and also “Delphi Developer” groups). It’s been a very hectic month so I fully understand that people are asking. So let’s look at where the project is at and where we are on the time-line.

For those that might not know, I decided to leave Embarcadero a couple of months ago. I will be working out may before I move on. I wanted to write about that myself in a clean fashion, but sadly the news broke on Facebook prematurely.

Long story short, I have been very fortunate to work at Embarcadero. I am not leaving because there is anything wrong or something like that. I was hired as SC for the EMEA regions, which basically made me the support and presenter for most of europe, parts of asia and the middle east. It’s been a great adventure, but ultimately I had to admit that my passion is coding and community work. Sales is a very important part of any company, but it’s not really my cup of tea; my passion has always been research and development.

So, come first of June and I start in a new position at RemObjects. A company that has deep roots with Delphi and C++ builder users – and a company that continues to produce a wealth of high-quality, high-performance frameworks for Delphi and C++ builder. RemObjects also has a strong focus on modern languages, and have a strong portfolio of new and exciting compilers and languages to offer. The Oxygene compiler should be no stranger to Delphi developers, a powerful object-pascal dialect that can target a variety of platforms and chipsets.

Since compiler technology and run-time systems has been my main focus for well over a decade now, I feel RemObjects is a better match.

Quartex Components

Quartex Components has been an officially registered Norwegian company for a while now, so perhaps not news. What is news is that it’s now directly connected with the development of the Quartex Media Desktop (codename “Amibian.js”). While Amibian.js is an open source endeavour, there will be both free and commercial products running on top of that platform. I have written at length about Cloud Forge in the past, so I wont re-hash that again. But 2020 will see a paradigm shift in how teams and companies approach software development.

quartex

Company logo professionally milled and on its way to my new office

I will also, once there is more time, continue to sell and support software license components.

Quartex Media Desktop

The “Amibian.js” project is moving along nicely. The deadline is Q4 2019, but im hoping to wrap up the core functionality before that. So we are on track and kicking ass 🙂

amibian_01

More and more elaborate functionality is being implemented for the desktop

Here is an overview of work done this month:

  • TSystemService application type has been created (node.js)
    • TApplication now holds IPC functions (inter process communication)
    • Running child processes + sending messages is now simplicity itself
    • Database drivers are 90% done. Delete() and DeleteTable() functionality needs to be implemented in a uniform way
  • Authentication is now a separate service
    • Service database layer is finished (using SQLite3 driver by default)
    • Authentication protocol has been designed
    • Server protocol and JSON message envelopes are done
    • Presently working on the client interface
  • LDEF bytecode assembler has been improved
    • Faster symbolic lookup
    • Smarter register recognition
    • Early support for stack-frames
    • Fixed bug in parser (comma-list parse)
  • QTX framework has seen a lot of work
    • Large parts of the RTL sub-strata has been implemented
    • UTF16 codec implemented
    • QTX versions of common controls:
      • TQTXButton
      • TQTXLabel
      • TQTXToolbar
        • TQTXToolButton
        • TQTXToolSeparator
        • TQTXToolElement
      • TQTXPanel
      • TQTXCheckBox
      • .. and much, much more
  • Desktop changes
    • Link Maker functionality has been added
    • Handshake process between desktop and child app now runs on a separate timer, ensuring better conformity and a more robust initialization
    • The Quartex Editor control has been optimized
      • All redraw calls are now synchronized
      • Canvas is created on demand, avoids flicker during initial redraw
      • Support for DEL key + behavior
      • Gutter is now rendered to an offscreen bitmap and blitted into the control’s canvas. The gutter is only fully rendered when cursor forces the view to change

I will continue to keep everyone up to date about the project. As you can understand, its a bit hectic right now so please be patient – it is turning into an EPIC environment!

VMWare: A Delphi developers best friend

March 3, 2019 1 comment

Full disclosure: I am not affiliated with any particular virtualization vendor of any sorts. The reason I picked VMWare was because their product was faster when I compared the various solutions. So feel free to replace the word VMWare with whatever virtualization software suits your needs.

On Delphi Developer we get new members and questions about Delphi and C++ builder every day. It’s grown into an awesome community where we help each other, do business, find jobs and even become personal friends.

A part of what we do in our community, is to tip each other about cool stuff. It doesn’t have to be directly bound to Delphi or code either; people have posted open source graphic programs, video editing, database designers – as long as its open source or freeware its a great thing (we have a strict policy of no piracy or illegal copying).

Today we got talking about VMWare and how its a great time saver. So here goes:

Virtualization

Virtualization is, simply put, a form of emulation. Back in the mid 90s emulators became hugely popular because for the first time in history – we had CPU’s powerful enough to emulate other computers at full speed. This was radical because up until that point, you needed special hardware to do that. You had also been limited to emulating legacy systems with no practical business value.

vmware

VmWare Workstation is an amazing piece of engineering

Emulation has always been there, even back in the 80s with 16 bit computers. But while it was technically possible, it was more a curiosity than something an office environment would benefit from (unless you used expensive compute boards). We had to wait until the late 90s to see commercial-grade x86 emulation hitting the market, with Virtuozzo releasing Parallels in 1997 and VMWare showing up around 1998. Both of these companies grew out of the data-center culture and academia.

It’s also worth noting that modern CPU’s now support virtualization on  hardware level, so when you are “virtualizing” Windows the machine code is not interpreted or JIT compiled – it runs on the same CPU as your real system.

Why does it matter

Virtualization is not just for data-centers and server-farms, it’s also for desktop use. My personal choice was VMWare because I felt their product performed better than the others. But in all fairness it’s been a few years since I compared between systems, so that might be different today.

53145702_10156048129355906_2019146241329332224_o

A screengrab of my desktop, here showing 3 virtual machines running. I have 64 gigabyte memory and these 3 virtual machines consume around 24 gigabytes and uses 17% of the Intel i7 CPU power during compile. It hardly registers on the CPU stats when idle.

VMWare Workstation is a desktop application available for Windows, Linux and OS X. And it allows me to create virtual machines, or “emulations” if you like. The result is that I can run multiple instances of Windows on a single PC. The virtual machines are all sandbox in large hard-disk files, and you have to install Windows or Linux into these virtual systems.

The bonus though is fantastic. Once you have installed an operating-system, you can copy it, move it, do partial cloning (only changes are isolated in new sandboxes) and much, much more. The cloning functionality is incredibly powerful, especially for a developer.

It also gives you something called snap-shot support. A snapshot is, like the word hints to, a copy of whatever state your virtual-machine is in at that point in time. This is a wonderful feature if you remember to use it properly. I try to take snapshots before I install anything, be it larger systems like Delphi, or just utility applications I download. Should something go wrong with the tools your work depends on — you can just roll back to a previous snapshot (!)

A great time saver

Updates to development tools are always awesome, but there are times when things can go wrong. But if you remember to take a snapshot before you install a program, or before you install a component package — should something go wrong, then rolling back to a clean point is reduced to a mouse click.

I mean, imagine you update your development tools right? Suddenly you realize that a component package your software depends on doesn’t work. If you have installed your devtools directly on the metal, you suddenly have a lot of time-consuming work to do:

  • Re-install your older devtools
  • Re-install your components and fix broken paths

That wont be a problem if you only have 2-3 packages, but I have hundreds of components install on my rig. Just getting my components working can take almost a full work-day, and I’m not exaggerating (!).

With VMWare, I just roll back to when all was fine, and go about my work like nothing happened.

I made a quick, slapdash video to demonstrate how easy VmWare makes my Delphi and JS development. If you are not using virtualization I hope this video at least makes it a bit clearer why so many do.

vmware_youtube

Click the image to watch the video on YouTube

Repository updates

February 25, 2019 2 comments

As most know by now, I was running a successful campaign on Patreon until recently. I know that some are happy with Patreon, but hopefully my experience will be a wakeup call about the total lack of rights you as a creator have – should Patreon decide they don’t understand what you are doing (which I can only presume was the case, because I was never given a reason at all). You can read more about my experience with Patreon by clicking here.

Setting up repositories

Having to manually build a package for each tier that I have backers for would be a disaster. It was time-consuming and repetitive enough to create packages on Patreon, and I don’t have time to reverse engineer Patreon either. Which I might do in the future and release as open-source just to give them a kick in the groin back.

To make it easier for my backers to get the code they want, I have isolated each project and sub-project in separate repositories on BitBucket. This covers Delphi, Smart Pascal, LDEF and everything else.

cloud_ripper

The CloudRipper architecture is coming along nicely. Here running on ODroid XU4

I’m just going to continue with the Tiers I originally made on Patreon, and use my blog as the news-center for everything. Since I tend to blog about things from a personal point of view, be it for Delphi, JavaScript or Smart Pascal — I doubt people will notice the difference.

So far the following repositories have been setup:

  • Amibian.js Server (Quartex Web OS)
  • Amibian.js Client
  • HexLicense
  • TextCraft (source-code parser for Delphi and Smart Pascal)
  • UAE.js (a fork of SAE, the JS implementation of UAE)

I need to clean up the server repository a bit, because right now it contains both the server-code and various sub projects. The LDEF assembler program for example, is also under that repository — and it belongs in its own repository as a unique sub-project.

The following repositories will be setup shortly:

  • Tweening library for Delphi and Smart Pascal
  • PixelRage graphics library
  • ByteRage bugger library
  • LDEF (containing both Delphi and Smart Pascal code)
  • LDEF Assembler

It’s been extremely busy days lately so I need to do some thinking about how we can best organize things. But rest assured that everyone that backs the project, or a particular tier, will get access to what they support.

Support and backing

I have been looking at various ways to do this, but since most backers have just said they want Paypal, I decided to go for that. So donations can be done directly via paypal. One of the new features in Paypal is repeated payments, so setting up a backer-plan should be easy enough. I am notified whenever someone gives a donation, so it’s pretty easy to follow-up on.

 

 

Updates used to be monthly, but with the changes they will be ad-hoc, meaning that I will commit directly. I do have local backups and a local git server, so for parts of the project the commits will be issued at the end of each month.

While all support is awesome, here are the tiers I used on Patreon:

  • $5 – “high-five”, im not a coder but I support the cause
  • $10 – Tweening animation library
  • $25 – License management and serial minting components
  • $35 – Rage libraries: 2 libraries for fast graphics and memory management
  • $45 – LDef assembler, virtual machine and debugger
  • $50 – Amibian.js (pre compiled) and Ragnarok client / server library
  • $100 – Amibian.js binaries, source and setup
  • $100+ All the above and pre-made disk images for ODroid XU4 and x86 on completion of the Amibian.js project (12 month timeline).

So to back the project like before, all you do is:

  1. Register with Bitbucket (free user account)
  2. Setup donation and inform me of your Bitbucket user-name
  3. I add you on BitBucket so you are granted access rights

Easy. Fast and reliable.

The QTX RTL

Those that have been following the Amibian.js project might have noticed that a fair bit of QTX units have appeared in the code? QTX is a run-time library compatible with Smart Mobile Studio and DWScript. Eventually the code that makes up Amibian.js will become a whole new RTL. This RTL has nothing to do with Smart Mobile Studio and ships with its own license.

Amigian_display

QTX approaches the DOM in more efficient way. Its faster, smaller and more powerful

Backers at $45 or beyond access to this code automatically. If you use Smart Mobile Studio then this is a must. It introduces a ton of classes that doesn’t exist in Smart Pascal, and also introduces a much faster and clean visual component framework.

If you want to develop visual applications using QTX and DWScript,  then that is OK,  providing the license is respected (LGPL, non commercial use).

Well, stay tuned for more info and news!

Quartex: Mali GPU glitches

February 20, 2019 Leave a comment

EDIT: I did further testing after this article was written, and believe the source of this to be about heat. Even with extra fans, running games like Tyrian (asm.js) that are extremely demanding, plus resizing a graphics intensive windows constantly, the temperature reached 71 degrees C very quickly. And this was with two cabinet fans helping the built-in fan to cool the device. It is thus not unthinkable that when running solo (no extra fans) that the kernel shut the device down to not cook the chipset. Which also explains why the device wont boot properly afterwards (the device is still hot).

Glitches

Something really strange is happening on Chrome and Firefox for ARM. JavaScript is not supposed to be able to take down a system, and in this case it’s neither an attempt as such either — yet for some reason I have managed to take down the ODroid XU4 with both Chrome and Firefox lately.

ODroid XU4

I guess I should lead with that I’m not able to replicate this on x86. One of the things I really love about the ODroid XU4 is that it’s affordable, powerful and probably the only SBC I have used that runs stable on the mali GPU. As you probably know I tested at least 10 different SBC’s back in 2018, and whenever there was a mali GPU involved, the product was either haunted by instabilities or lacked drivers all together.

amibian

Since the codebase for Chrome (and I presume Firefox) is ultimately the same between platforms, it leaves a question-mark about the ODroid. It is by far the most stable SBC I have tested so far (except for the PI, which is sadly underpowered for this task), but stable doesn’t mean flawless. And to be honest, Amibian.js is pushing web tech to the very limits.

Not Mali again

The reason I suspect the mali to be the culprit behind all this, is because the “bug” if we can call it that, happens exclusively during resize. So if there is a lot going on inside a desktop-window, you can sometimes provoke the ODroid to cold-crash and reboot. You actually have to power the board down and switch it back on for it to boot properly.

50431451_10155954273110906_8776790185049325568_n

Cloudripper ~ 5x ODroid XU4 [40 cores] in a PICO 5h cube

The resize and moving of windows uses CSS transformation, which in modern browsers makes use of the GPU. Chrome talks directly with OpenGL (or glES), so the operations are proxied through that. And again, since OpenGL is pretty rock solid elsewhere, we are only left with one common denominator: the mali GPU.

The challenge is that there is no way to debug or catch this error, because when it occurs the whole system literally goes down. There is no exception thrown, nor is the browser process terminated (not even a log entry, so it’s a clean-cut) — the system reboots on the spot. Since it fails on reboot when opening X (setting a screen-mode) I again point the finger at the GPU. Somehow a flag or lock survives the cold-reboot and that’s why you have to manually switch it off and on again.

This is the exact problem that made the NanoPI Fire useless. It only shipped with Android embedded drivers. The X drivers could hardly open a display without crashing. Such a waste of a good cpu.

x86 as head

ODroid is perfect for a low-cost Amibian.js experience, but I was unsure if it would handle the payload. Interestingly it handles it just fine and even with a high-speed action game running + background tasks we are not using 50% of the CPU even.

Ram is holding up too, with memory consumption while running Tyrian + having a few graphics viewers open, is at a reasonable 700 mb (of 2 gigabyte in total).

51398321_10155998598505906_8984850199142727680_o

Tyrian jogs along at 45 fps ~ that is not bad for a $45 SBC

Right now this strange error is rare, but if it continues or grows into a problem (chrome is hardly useable at all, only firefox) then I have no option than to replace the master sbc in the cluster with something else. The x86 UP board is more than capable, but it would be a shame to break the price range because of that (excuse my language) crap mali GPU. I honestly don’t understand why board makers insist on using a mali. Every board that has a mali is haunted by problems and get poor reviews.

It will be exciting to check out the dragonboard, although I fear 1Gb memory will not be enough for smooth operation. Not without a sata interface and a good swap-file.

Android and Delphi

One alternative is to switch to Android and use Delphi to code a custom Chromium Embedded webview. I am hoping to avoid the overhead of Android, but Delphi would definitively be a bonus with Android embedded (“Android of things”).

We will see.

Leaving Patreon: Developers be warned

February 17, 2019 4 comments

As a person I’m quite optimistic. I like to think the glass is half-full rather than half-empty. I have spent over a decade building up a thriving Delphi and C++ builder community on social media, I have built up a rich creative community for node and JavaScript on the side — not to mention retro computing, embedded tech and IOT. For better or for worse I think most developers in the Embarcadero camp have heard my name or engage in one of the 12 groups I manage around the world on a daily basis. It’s been hard work but man, it’s been worth every minute. We have so much fun and I get to meet awesome coders on a daily basis. It’s become an intrinsic part of my life.

I have been extremely fortunate in that despite my disadvantage, a spine injury in 2012 – not to mention being situated in Norway rather than the united states; despite these obstacles to overcome I work for a great American company, and I get to socialize and have friends all over the planet.

The global village is the concept, or philosophy, that technology makes it possible no-matter where you live, to connect and be a part of something bigger. You don’t have to be a startup in the san-francisco area to work with the latest tech. Sure a commute from Burlingame to Redwood beats a 14 hour flight from Norway any day of the week — but that’s the whole idea: we have Skype now, and Slack and Github; you don’t have to physically be on location to be a part of a great company. The only requirement is that you make yourself relevant to your field of expertise.

Patreon, a digital talent agency

Patreon is a service that grew straight out of the global village. If the world is just one place, one great big family of human beings with great ideas, then where is the digital stage that helps nurturing these individuals? I mean, you can have a genius kid living in poverty in Timbuktu that could crack a mathematical problem on the other side of the globe. The next musical prodigy could be living in a loft in Germany, but his or her voice will never be heard unless it’s recognized and given positive feedback.

“The irony is that Patreon doesn’t even pass their own safety tests. That should make you think twice about their operation”

My examples are extremes I agree, most people on Patreon are like me, creative but absolutely not cracking math problems for Nasa; nor am I singing a duet with Bono any time soon. But that’s the fun thing about the world – namely that all things have value when put in the correct context. Life is about combinations, and you just have to find one that works for you.

village

The global village, the idea of unity through diversity

The global village is this wonderful idea that we can use technology to transcend the limitations the world oppose on us, be they nationality, color, gender or location. Good solutions know no bounds and manifests wherever a mind welcomes it. Perhaps a somewhat romantic idea, if not naive, but it seems the only reasonable solution given the rapid changes we face as a species.

In my case, I love to make software components in my spare time. My day job is packed and I couldn’t squeeze in more work during the weekdays if I wanted to, so I only have a couple of hours after-work and the weekends to “do my thing”. So being a total geek I relax by making components. Some play chess, the guitar or whatever — I relax by coding something useful.

Obviously “code components” are completely useless to anyone who is not a software developer. The relevance is further clipped by the programming-language they are written for, and ultimately the functionality they provide. Patreon for me was a way to finance the evolution of these components. A way of self motivating myself to keep them up to date and available.

I also put a larger project on Patreon, namely the cloud desktop system people know as “Amibian.js” or “Quartex Web OS”. Amibian being the nickname, or codename.

Patreon seemed like the perfect match. I could take these seemingly unrelated topics, Delphi and C++ builder specific components and a cloud architecture, and assign each component and project to separate “tiers” that the audience could pick from. This was great! People could now subscribe to the tier’s they wanted, and would be notified whenever there was an update or new features. And I could respond to service messages in one place.

The Tier System

The thing about software is that it’s not maintained on infinite repeat. You don’t fix a component that is working. And you don’t issue updates unless you have fixed bugs or added new functionality. A software subscription secures a customer access to all and any updates, with a guarantee of X number of updates a year. And equally important, that they can get help if they are stuck.

“when you are shut down without so much as an explanation, with nothing but positive feedback, zero refunds and over 1682 people actively following the progress — that is utterly unacceptable behavior”

I set a relatively low number of guaranteed updates per year for the components (4). The things that would see the most updates were the Rage Libraries (PixelRage and ByteRage) and Amibian.js, but not until Q3 when all the modules would come together as a greater whole — something my backers are aware of and have never had a problem with.

Amigian_display

Amibian.js running on ODroid XU4, a $45 single board computer

The tiers I ended up with was:

  • $5 – “high-five”, im not a coder but I support the cause
  • $10 – Tweening animation library
  • $25 – License management and serial minting components
  • $35 – Rage libraries: 2 libraries for fast graphics and memory management
  • $45 – LDef assembler, virtual machine and debugger
  • $50 – Amibian.js (pre compiled) and Ragnarok client / server library
  • $100 – Amibian.js binaries, source and setup
  • $100+ All the above and pre-made disk images for ODroid XU4 and x86 on completion of the Amibian.js project (12 month timeline).

Note: Each tier covers everything before them. So if you pick the $35 tier, that also includes access to the license management system and the animation library.

As you can see, the tier-system that is intrinsic to Patreon, solves the software subscription model elegantly. After all, it would be unreasonable to demand $100 a month for a small component like the Tweening library. A programmer that just needs that library and nothing else shouldnt have to pay for anything else.

Here is a visual representation, showing graphically why my tiers are organized as they are, and how they all fit into a greater whole:

tier_dependencies

The server-side aspect of the architecture would take days to document, but a general overview of the micro-service architecture is fairly easy to understand:

tier_dependencies2

Each of the tiers were picked because they represent key aspects of what we need to create a visually pleasing, fast and reliable, distributed (each part running on separate machines or boards) cloud eco-system. Supporters can just get the parts they need, or support the bigger project. Everyone get’s what they want – all is well.

The thing some people don’t grasp, is that you are not getting something to just put on Amazon or Azure, you are getting your own Amazon or Azure – with source code! You are not getting services, you are getting the actual code that allows YOU to set up your own services. Anyone with a server can become a service provider and offer both hosting and software access. And they can expand on this without having to ask permission or pay through the nose.

So it’s a little bit bigger than first meets the eye.

I Move In Mysterious Ways ..

Roughly 3 weeks ago I was busy preparing the monthly updates.

Since each tier is separate but also covers everything before it (like explained above) I have to prepare a set of inclusive updates. The good news is that I only have to do this once and then add it as an attachment to my posts. Once added I can check of all the backers in that tier. I don’t have to manually email each backer, physically copy my songs or creations onto CD and send it – we live in the digital age as members of the global village. Or so i thought.

So I published two of the minor cases first: the full HTML5 assembly program, that can be run both inside Amibian.js as a hosted application — or as a solo program directly in the browser. So here people can write machine-code in the browser, assemble it to bytecodes, run the code, inspect registers, disassemble the bytecodes and all the normal stuff you expect from an assembler.

This update was special because the program contained the IPC (inter process communication) layer that developers use to make their programs talk to the desktop. So for developers looking to make their own web programs access the filesystem, open dialogs (normal system features), that code was quite important to get!

tier_posts

Although published, none of my backers could see them due to the suspended status

The second post was a free addition, the QTX library which is an open-source RTL (run time library) compatible with the Smart Pascal Compiler. While not critical at this juncture, several of my backers use Smart Mobile Studio, and for them to get access to a whole new RTL that can be used for open-source, is very valuable indeed.

I was just about to compress the Amibian.js source-code and binaries when I got a message on Facebook by a backer:

“Dude, your Patreon is shut down, what is happening?”

What? hang on let me check i replied, and rushed into Patreon where the following header greeted me:

tier_header

What the hell Patreon? I figured there must be some misunderstanding and that perhaps I missed an email or something that needed attention. I get close to 50 emails a day (literally) so it does happen that I miss one. I also check my spam folder regularly in case my google filters have been careless and flagged a serious email as spam. But there was nothing. Not a word.

Ok, so let’s check the page feedback, has there been any complaints? Perhaps a backer has misunderstood something and I need to clear that up? But nope. I had nothing but positive feedback and not even a single refund request.  In fact the Amibian.js group on Facebook has grown to 1,662 members. Which shows that the project itself holds considerable interest outside software development circles.

Well, let’s get on this quickly I thought, so I rushed off an email asking why Patreon would do such a thing? My entire Patreon page was visibly marked with the above banner, so my backers never even saw the updates I had issued.

Instead, the impression people would get, was that I was involved in something so devious that it demanded my account to be suspended. Talk about shooting first and asking later. I have never in my life seen such behavior from a company anywhere, especially not in the united states; Americans don’t take kindly to companies behaving like bullies.

Just Contact Support, If You Can Find Them

To make a long story short it took over a week before Patreon replied to my emails. I sent a total of 3 emails asking what on earth would have prompted them to shut down a successful campaign. And how they found it necessary to slander the project without even informing me of the problem. Surely a phone call could have sorted this up in minutes? Where I come from you pick up the phone or get in contact with people before you flag them in public.

patreon

Sounds great, sadly it’s pure fiction

The response I got was that “some mysterious activity had been reported on my page”, and that they wanted my name, address, phone number and credit card (4 last digits). Which I found funny because with the exception of credit-card details, I always put my name, address, phone numbers and email etc. at the head of my letters.

I’m not a 16-year-old kid working out of a garage, im a 46-year-old established software developer that have worked as a professional for close to 3 decades. Unlike the present generation I moved into my first apartment when I was 16, and was working as an author for various tech magazines by the time I was 17. I also finished college at the same time and went on to higher-education (2 years electrical engineering, 3 years arts and media, six years at the university in oslo, followed by 4 years of computer science and then certifications). The focus being, that Patreon is used to dealing with young creators that will go along with things that grown men would not accept.

But what really piss me off, was that they never even bothered to explain what this “mysterious behavior” actually was? I write about code, clustering, Delphi, JavaScript and bytecodes for christ sake. I might have published updates and code wearing a hoodie at one point, in a darken room, listening to Enigma.. but honestly: there is not enough mystery in my life to cover an episode of Scooby-Doo.

Either way, I provided the information they wanted and expected the problem to be resolved asap. Two days at themost. Maybe three, but that was pushing it.

It’s now close to 3 weeks since this ridiculous temporary suspension occurred, and neither have I been given any explanation to what I have done, nor have they removed the ban on the content. I must have read their guidelines 100 times by now, but given the nature of their ruling (which are more than reasonable), I can’t see that I have violated a single one:

  • No pornography and adult content
  • No hate speech against minorities or forms of religious extremism
  • No piracy or spreading copyrighted material
  • No stealing from backers

Let’s go over them one by one shall we?

Pornography and adult content

Seriously? I don’t have time to loaf around glaring at naked women (i’m a geek, I look weird enough as it is), and after 46 years on this planet I know what a woman looks like nude from every possible angle; I don’t need to run around like a retard posting pictures of body parts. And if you are talking about me — good lord is there a marked for hobbits? Surely the world has enough on it’s plate. Sorry, never been huge on porn.

And for the record, porn is for teenagers and singles. The moment you love someone deeply, the moment you have children together — it changes you profoundly. You get a bond to your wife or girlfriend that makes you not want to be with others. Not all men are into smut, some of us are invested more deeply in a relationship.

Hate speech and religious extremism

Hm, that’s a tough one (sigh). Did you know that one of my best friends is so gay – that he began to speculated that he actually was a liquid? He makes me laugh so bad and he’s probably the best human being I have ever met. I actually went with him on Pride last year, not because i’m gay but because he needed someone to hold the other side of the banner. That’s what friends do. Besides, I looked awesome, what can I say.

As for religion I am a registered Tibetan Buddhist. I believe in fluffy pillows, comfy robes, mother nature and quite frankly I find the world inside us far more interesting than the mess outside. You cant be extreme in Buddhism: “Be kind now, or ill hug you until you weep the tears of compassion!”. Buddhism sucks as an extreme doctrine.

So I’m going to go out on a limb and say nuuuu to both.

Piracy and copyrighted material

Eh, I’m kinda writing the software from scratch before your eyes (including the run-time-library for the compiler), so as far as worthy challenges go, piracy would be the opposite. I am a huge fan of classical operating-systems though, like the Amiga; But unlike most people I actually took the time to ask permission to use a OS4 inspired CSS theme-file.

asana

The Amibian.js project is well organized and I have worked systematically through a well planned architecture. This is not some slap-dash project made for a quick buck

Most people just create a theme-file and don’t bother to ask. I did, and Trevor Dickinson was totally cool about it. And not a single byte has been taken or stolen from anyone. The default theme file is inspired by Amiga OS 4.1, but the thing is: the icons are all freeware. Mason, the guy that did the OS icons, have released large sets of icons into GPL. There is also a website called OS4Depot where people publish icons and backdrops that are free for all.

So if this “mysterious activity” is me posting a picture of a picture (not a typo) of an obscure yet loved operating-system, rest assured that it’s not violating anyone.

Stealing from backers

That they even include this as a point is just monumental. Patreon is a service established to make that impossible (sigh); meaning that the time-frame where you deliver updates or whatever – and the time when the payout is delivered, that is the window where backers can file a complaint or demand a refund.

And yes, complaints on fraud would indeed (and should!) flag the account as potentially dubious — but again, I have not a single complaint. Not even a refund request, which I believe is pretty uncommon.

And even if this was the case, shutting down an account without so much as a dialog in 2019? Who the hell becomes a thief for 600 dollars? Im not some kid in a garage, I make twice that a day as a consultant in Oslo, why the heck would I setup a public account in the US, only to run off with 600 bucks! I have standing offers for projects continuously, I havent applied for a job since the 90s – so if I needed some extra money I would have taken a side project.

I even posted to let my backers know I had a cold last month just to make sure everyone knew in case I was unavailable for a couple of days. Truly the tell-tell sign of a criminal mastermind if I ever saw one ..

tier_refunds

Sorry Patreon, but your behavior is unacceptable

Hopefully your experience with Patreon has not been like mine. They spent somewhere in the range of 5 weeks just to register me, while friends of mine in the US was up and running in less than 2 days.

We are now 3 weeks into a temporary suspension, which means that most of my backers will run out of patience and just leave. It sends a signal of being whimsical about other people’s trust, and that people take a risk if they back my project.

At this point it doesn’t matter that none of these thoughts are true, because they are thoughts that anyone would think when a project remains flagged for so long.

What should scare you as a creator with Patreon though, is that they can do this to anyone. There is nothing you can do, neither to prove your innocence or sort out a misunderstanding — because you are not even told what you allegedly have done wrong. I also find it alarming that Patreon actually doesn’t have a phone-number listed, nor do they have offices you can call or reach out to.

The irony is that Patreon doesn’t even pass their own safety tests. That should make you think twice about their operation. I had heard the rumors about them, but I honestly did not believe a company could operate like this in our day and age. Especially not in the united states. It undermines the whole spirit of US as a technological hub. No wonder people are setting up shop in China instead, if this is how they are treated in the valley.

After this long, and the damage they have caused, I have no option than to inform my backers to terminate their pledges. I will have to relocate my project to a host that has more experience with software development, and who treats human beings with common decency and respect.

If I by accident had violated any of their guidelines, although I cannot see how I could have, I have no problem taking responsibility. But when you are shut down without so much as an explanation, with nothing but positive feedback, zero refunds and over 1682 people actively following the progress — that is utterly unacceptable.

It is a great shame. Patreon symbolized, for a short time, that the global village had matured into more than an idea. But I categorically refuse to be treated like this and find their modus-operandi insulting.

Stay Well Clear

If you as a developer have a chance to set up shop elsewhere, then I urge you to do so. And make sure your host have common infrastructure such as a phone number. Patreon have taken the art of avoiding direct contact to a whole new level. It is absolutely mind-boggling.

I honestly don’t think Patreon understands software development at all. Many have voiced more sinister motives for my shutdown, since the project obviously is a threat to various companies. But I don’t believe in conspiracies. Although, if Patreon does this to enough creators on interval, the interest rates from holding the assets would be substantial.

It could be that the popularity of the project grew so fast that it was picked up as a statistical anomaly, but surely that should be a good thing? Not to mention a potential case study Patreon could have used as a success story? I mean, Amibian.js didn’t get up and running until october, so stopping a project 5 months into a 12 month timeline makes absolutely no sense. Unless someone did this on purpose.

Either way, this has been a terrible experience and I truly hope Patreon get’s their act together. They could have resolved this with a phone-call, yet chose to let it fester for almost a month.

Their loss.

Quartex Web OS: A cloud OS in takes form

January 19, 2019 Leave a comment

It’s been a while since I’ve posted now. I have 3 articles in escrow, and every time I think I will finish them, I end up writing more. But yes, more Delphi articles is coming and I have lined up both components and rich code that everyone will be happy about.

Please look before shooting

Before we dig into the new stuff, I want to clear up a misconception. We programmers often forget that not everyone knows what we do, and we take it for granted that everyone will instantly understand something we talk about. Which is rarely the case.

I have noticed that quite a few have misjudged the project radically, thinking that the first version (cloud ripper) is just a toy, a mock desktop or even worse: just a remake of a legacy system that “has no role in modern computing”.

It is true that I have taken more than a little from Amiga OS in terms of architecture, but I have exclusively taken ideas that are good and works well under the ASYNC execution model. I have also replicated the way the filesystem is organized, things like REXX (which was added to OS X in 2015), the menu system – these are indeed built on how Amiga OS did things. The same can be said about library functions. Not because they are old, but because they make sense. Many of the functions appear in other systems too, like GTK on Linux and WinAPI for Windows. There are only so many ways to open a window, change the title, define scrollbars and execute processes.

kiosk-systems

Kiosk systems like this are great targets for the Quartex Web OS

While there are clear architectural aspects taken from older systems, doesn’t mean that the system itself is old in any way. This system is designed to run as WebAssembly, ASM.js and vanilla Javascript – which is ASYNC by nature. It is designed to run and share payload over several machines, not a single outdated CPU and chipset. You have swarm based task solving – which is quite cutting edge if I might say so. None of these things were invented back in the day.

Some have also asked why this is even needed. Well, let me give you a simple use case.

One of my customers is doing work for Jensen, a Danish producer of IT hardware. They make mostly routers, wifi usb dongles and similar devices. But like many hardware vendors their web interface leaves a lot to be desires. Router web interfaces are usually quite annoying and poorly written. Something that should have taken 5 minutes can end up taking 30 just because the design of the interface is rubbish.

With my solution these vendors will be able to drop a whole infrastructure into their products; a infrastructure that provides all the things they need to quickly build a great control panel and router interface. Things like file system mapping, being able to store data to the filesystem through an established websocket protocol; all of it wrapped up in a simple but powerful API. Their settings and features can be represented as programs, which run in windows that are intuitively styled and easy to understand. They will also cut development time dramatically by calling the Quartex Soft-Kernel, rather than having to re-invent everything from scratch.

That is just a tiny, tiny use-case where the desktop and services makes perfect sense. But also keep in mind that the same system can scale up to a 1000 instance Amazon supercomputer if you need to, providing software for your offices and development teams.

In 8 months the desktop is complete (probably before) and I start building the first purely web powered software development toolchain. Everything has been transformed into Javascript (as in compilers, linkers – the whole lot). Both freepascal, clang c/c++ and much more. And developers will be able to login and start producing applications out of the box. The fact that the entire system is chipset and platform independent is quite unique. People tend to use native code behind a facade of html5. Not here. Here you have over 4000 classes, 800.000 lines of code just for the desktop client, looking back at you.

Hopefully this has shed some lights on the project, and people will stop looking at this as “old junk”. As a person who loves older computers, Amiga especially, I am quite frankly astounded by the ignorance regarding that platform. A juiced up 30 year old Amiga will give any modern computer a run for it’s money when it comes to ease of use, quality software and pure productivity. 10 years before Windows even existed, europeans enjoyed a colorful, window based desktop with full multitasking. When we had to switch to PC it was like going back to the 1500’s in terms of functionality – and it wasnt until Windows 7 that Microsoft caught up with Commodore. So if I have managed to get over even 1% of the spirit in that machine – then I will be very happy indeed.

But to limit a clustered, 40 CPU core architecture using modern, off-the-shelves parts, a multitude of node services to “old junk” is nothing short of an intellectual emergency. Please read, digest and look more closely before passing judgement.

Right then, so what’s new?

48365835_10155890849180906_6431235229611982848_n

The Quartex “Cloud Ripper”

Where to begin! Like mentioned in my previous post Amibian.js is a cluster system. As such the project now has its first real hardware sorted! I have gone for a 5 x ODroid XU4 model, neatly tucked inside a PICO 5H case. The budget was set at USD 400, but with shipping and taxes it ended up costing around USD 600. But that is not a bad price for the firepower you get (40 CPU cores, 20 GPU cores and 16 Gb Ram), the ODroid is a powerful, stable and reliable ARM SBC (single board computer). In benchmarks the Raspberry PI 3b scored 830 Dhrystones, the ODroid scored 5500 Dhrystones. And my architecture use five of them, so this is a $600 super-computer built using off the shelves part.

The back-end server has had several bugs fixed, especially the problems with path’s and databases. You can now edit the settings.ini file and tell the system where the database should be created or accessed from, you can set the port for the server, if it should use SSL + Secure WebSocket,  or ordinary HTTP + Websocket.

50511885_10155952491120906_1059229155276619776_o

40 ARM CPU cores, that is a lot of firepower for USD 200 !

I am also ditching the TW3NodeFileSystem driver for server logic and using ordinary node.js calls there. The TW3NodeFileSystem driver is mounted as you perform a login – and it acts as a sandbox, mounting your folder as a device (and making sure you can’t ever touch files outside your “home” server folder). We still need to implement a proper UNIX directory parser, but that is easy enough.

Quartex Pascal

Yes, I have picked up Quartex Pascal again, which originally started in 2014. I have started writing a new RTL for DWScript which is an alternative to Smart Mobile Studio. It is different from the Smart RTL and is closer to FMX than VCL.

Eventually the Quartex Web OS and all its services will compile without code from Smart Mobile Studio.

Hosted applications, messages and our soft-kernel

The biggest news, which is also the most tricky to get right, is getting hosted applications (applications are hosted in IFrame containers) to communicate with the desktop. As you probably know browsers have rigid security measures, and the rules for threads (web workers) and separate processes (frames) are severe to say the least.

50407351_795409364151096_4870092648481816576_n

The LDEF assembler is the first application to grace the system

A secondary application hosted in a frame has absolutely no access to the rest of the DOM. Meaning that the code has no way of calling functions or manipulating elements outside its own DOM in the frame container. This is a good system because we don’t want rouge applications causing havoc.

The only way an application can talk to the desktop is through messages. And while this sounds easy, remember: we are doing this as a solid system, not just slapping something together.

  • After loading a hosted application, the desktop will send a handshake request. It will do this on interval until the application accepts.
  • When the application replies with a handshake message, the desktop sends a special message-channel object to the app. All communication with the desktop must happen on that secure channel.
  • With the channel obtained, the application has to provide the application manifest file. This is a special INI-File containing information about the program, including access rights. None of the soft-kernel API functions will execute until a valid manifest-file has been delivered.
  • Once the manifest has been sent and accepted, the hosted application is free to call the soft-kernel functions.

The above might sound simple but it includes several sub technologies to be in place first:

  • Call Stack: a class that keep track of sent messages and a callback. When a response arrives it will execute the correct callback to deliver the response. This is a kind of “promises” engine for message delivery.
  • Message factory, matches message-data to the correct message class, creates the instance and de-serialize the data automatically for you
  • Message dispatcher: Allows you to register a message with a handler procedure. When a message arrives the dispatcher calls the message-factory, then calls the correct handler.
  • Base64 Encoding on byte-array, stream and buffer level (does not exist in either node.js or JavaScript in general)
  • String to UTF8 Byte-Array encoding
  • UTF8 Byte-Array to String encoding
  • escape and unescape for byte-array, stream and buffer
  • URI-encoder for byte-array, stream and buffer

But that was just the beginning, I also had to introduce an object that I have been dreading to even start on, namely the “process” class. The process is not just a simple reference to the frame container, it has to keep track of the websocket endpoint, application manifest, error handling, message routing and much more.

50077678_10155951521540906_6068161951656050688_o

CLANG compiled to webassembly, meaning we can now compile proper C/C++ in the browser

Since Amibian.js supports not just JavaScript, but also bytecode applications – the process object also contains the LDEF runtime engine; not to mention all the system resources a process can own.

The cool part is that things work exactly like I planned! There is plenty of room to optimize, but all in all the architecture is sound. And it was quite a hallelujah moment when the first API call went through at 00:00 19.01.2019! A call to SetWindowTitle() where the hosted application set the caption of its main-window purely via code. Cross domain communication at it’s very best.

The LDEF Assembler

Yes LDEF Bytecodes are fantastic, and the first program I have made is a traditional assembler. I went all in and implemented a full text-editor to get better control, and also to get rid of the ACE code editor, which was a massive dependency. So glad we got rid of that.

So now you can write assembly code, assemble it, run it, dis-assemble it and even dump the bytecodes to the window. You will be able to save the bytecodes to disk by the end of this weekend, and then run the bytecode programs from shell or the desktop. So we are really making progress here.

49938355_1169526123220996_502291013608407040_o

A good shell / pipe infrastructure is the key to a powerful desktop

LDEF is the bytecode system that will be used to build high-level languages like Basic and Pascal. Since Freepascal is now able to compile itself to JavaScript I will naturally add that to the IDE next fall; the same is true for CLANG which has compiled itself to WebAssembly — and who generates webassembly.

So C/C++ and object pascal are already working and waiting for the IDE.

LDEF is a grander system though, because libraries can be loaded by Delphi, C++ builder, C# or whatever you fancy – and used. It can be post-processed to real machine code, or converted to pure WebAssembly. It holds much wider scope than stack machines like CLR and Java, and its more natural for assembly programmers – because it’s based on real CPU’s. It’s a register based virtual machine, not a stack-machine.

More?

Tons, but you have to visit my patreon page to keep track. I try to publish as much as possible there rather than here. I post a bit on both, but the proper channel for Amibian.js (or “Quartex Web OS” as its official name is) will always be Patreon.

50108015_314551789176307_8213345524409958400_n

The picture viewer now has momentum scrolling in full-mode.

Also, fixed more bugs in the Smart RTL than I can count, and re-made window movement. Window movement now uses the GPU, so they are silky smooth everywhere. Resize will be optimized next, then you can’t really tell it’s not native code at all.

Delphi Component updates

Yes Delphi is also a huge part of the Patreon project, and you will be happy to hear that the form designer (which shares a codebase with the graphics application components) have seen more work!

You can check out some of the changes to the form-designer here:

These changes will be in the january update (end of month) together with all the changes to Amibian.js, HexLicense, Tween library and all the rest 🙂

Cheers!