Archive
QuartexDeveloper.com is now active
It’s taken a while but Quartex Pascal now has it’s own website and forum. You can visit QuartexDeveveloper.com and check it out.
The SSL certificates are being applied within 72hrs according to the host, so don’t be alarmed that it shows up under HTTP rather than HTTPS right now – that is just temporary.
Up until now we have operated with a mix of donations and Patreon to help fund the project, but obviously that model doesn’t scale very well. After some debate and polls on the Facebook group I have landed on a new model.
Funding and access model
Starting with the release of version 1.0, which is just around the corner – the model will be as such:
- Backing and support will be handled solely through Patreon
- Facebook group will become open for all
- Patreon tiers will be modified to reflect new model
- Main activity and news will shift to our website, quartexdeveloper.com
- Community build will be available from our website
- Commercial license will also be available from our website
So to sum up, the following 3 options are available:
- Back the project on Patreon, full access to the latest and greatest 24/7
- Community edition, free for educational institutions and open-source projects (non commercial)
- Commercial license is for those that don’t want to back the project on a monthly basis, but instead use the community edition in a professional capacity for commercial work.
With the community edition available, why should anyone bother to back the project you might ask? Well, the public builds will by consequence be behind the latest, bleeding edge builds since the community edition is only updated on minor or major version increments (e.g. when version changes from 1.0 to 1.1). Users who back the project via Patreon will have instant access to new documentation, new packages with visual components, new project templates, RTL fixes and patches as they are released. These things will eventually trickle down to the community edition through version increments, but there is a natural delay involved.
This is how most modern crowd funded projects operate, with LTS builds (long term support) easily available while the latest cutting edge builds are backers only. Documentation, fixes and updates to components, new component packages, hotfixes and so on – is the incentive for backing the project.
This is the only way to keep the ball rolling without a major company backing day to day development, we have to get creative and work with what we got. Projects like Mono C# had the luxury of two major Linux distribution companies backing them, enabling Miguel de Icaza to work full time on the codebase. I must admit I was hoping Embarcadero would have stepped in by now, but either way we will get it done.
Onwards!
Quartex Pascal Build 13b ships
While it can come across as disingenuous, I frickin love this project! As a developer yourself you know that feeling, when you manage to unify various aspects of your program, so that it all fits just perfectly. And the way I implemented file-handling and editors is within that sweet spot.
What is new?
It’s been a couple of weeks since I posted here last, so the list of changes will be quite dramatic. I think its best to focus on the highlights or this post would become very long!
Ironwood license management
Up until 2018 one of my products was a component package called HexLicense. This is a component package for Delphi that provides serial number validation, license handling and (most importantly) serial number minting. The HexLicense components were sold commercially until 2018 when I took them off the market and open-sourced (access was via Patreon. It is now moved to the Quartex Pascal project instead).
Im not going to go into how difficult it is to produce thousands of distinctly different serial numbers based on seed data, but it’s no walk in the park.
The final implementation I made for license minting and validation, was called Ironwood. It took the engine behind HexLicense and took it to a completely new level, incorporating both obfuscation and number modulation.
Needless to say, Ironwood is now a part of the Quartex Pascal RTL. To make it easier to work with the IDE has a nice utility for generating license-numbers, loading and saving keys, exporting license number batches – and much more.
There is also a ready-to-rock node.js application that can generate keys from the command-line (which is good to invoke from a server or service, so that it executes as a separate process).
HTML structure provider
The IDE has a very clean internal architecture, where the actual work is isolated in a set of easy to understand classes. One of these classes is called a TIDEAstProvider class. This is a class whose job it is to parse and otherwise work with whatever content an editor has, and deliver symbolic information that can be displayed in the file-structure treeview.
Obviously we have an object pascal provider, which will quickly compile and generate an AST very quickly in memory. This is used to power both the structure treeview and the code-suggestion.
Next, we have the exact same provider for JavaScript. So when you open a JavaScript file, the file will be processed to produce an AST, and the symbol information will be displayed exactly like your object pascal is. So behavior between these are identical.
We now also have a HTML provider, with a CSS provider on the way. The HTML provider is still in its infancy, but its flexible enough to represent a good foundation to work with. So I will no doubt return to this task later to smarten the provider logic up, and better handle un-valid HTML and CSS.
Code suggestion
Code suggestion is a pretty standard function these days. We have had support for this under Object Pascal for a while now in the IDE (with JavaScript on the way).
Note: the code suggestion-box is un-styled at this point. Custom painting will be added once the core functionality is done.
Code suggestion for HTML is now in place too. It needs a bit of polish since the rules for HTML are wildly different from a programming language, but common behavior like TAG suggestion is there — with attributes, properties and events to follow.
So even if you are not an object pascal developer, the IDE should be nice to work with for traditional JavaScript / HTML code.
Form Recognition
While we cannot activate the form-designer just yet, since we need more AST functionality to extract things like class properties and attributes “live” to be able to do that properly — we are getting really close to that milestone.
The IDE however now recognize form files, so if your unit has an accompanying DFM file, the IDE is smart enough to open up a form-page. Form pages are different from ordinary pascal pages, since they also have the form designer control on a sub-tab. More or less identical to Delphi and Lazarus.
It is going to be so nice to get the form-designer activated. Especially the stack-based layout, which makes scalable, dynamic layout easy to create and work with.
The QTX RTL also supports orientation awareness as a part of the visual component system. One of the first things you will notice when exploring the code, is that ReSize() ships in an Orientation parameter, so you can adjust your layout accordingly.
Help and documentation inside the IDE
The IDE now has a PDF viewer with search functionality built-in. So when you click on Help and Documentation, a tab which shows the documentation PDF opens. This makes it easy to read, learn and find the information you need fast.
Well, that was a brief overview of what has changed since last time!
Next update is, as always, the weekends. We tend to land on sundays for new binaries, but do issue hotfixes in the evenings (weekdays) if something critical shows up.
Come join the fun!
Want to support the project? All financial backers that donates $100+ get their name in the product, access to the full IDE source-code on completion, and access to the Quartex Media Desktop system (which is a complete web desktop with a clustered back-end, compiled to JavaScript and running on node.js. Portable, platform and chipset independent, and very powerful).
A smaller sum monthly is also welcome. The project would not exist without members backing it with $100, $200 etc every month. This both motivates and helps me allocate hours for continuous work.
When the IDE is finished you will also have access to the IDE source-code through special dispensation. Backers have rights since they have helped create the project.
All donations are welcome, both large and small. But donations over $100, especially reoccurring, is what drives this project forward.
Remember to send me a message on Facebook so I can add you to the Admin group: https://www.facebook.com/quartexnor/
Quartex Pascal, status
With Quartex Pascal development at full steam and a dedicated Facebook group for the backers – It’s not often that I post updates here on my blog. One of the benefits of being a backer is that you have direct access to the latest builds, and also that you take part in the dialog on the group.
Be that as it may, here are some of the news happening with Quartex Pascal!
What’s new?
Quite a bit has happened since my last blog post. The IDE is coming together piece by piece, and at the moment i’m focusing on helper functionality for the AST (abstract symbol tree).
As you no doubt know, when a program is compiled it’s first parsed and converted into objects in memory. So every inch of your program becomes an elaborate tree-structure. This structure is what is commonly called the AST – and it is the raw material if you will, that code is generated from. In our case we don’t produce machine code, but rather JavaScript.
As you can imagine such an AST is quite complex. It has to be able to represent all the nuances of Object Pascal, as well as simplify finding information about everything. Every datatype, every record, class or complex structure, every field, expression — it all exists within the AST.
Without code to quickly traverse and work with the AST, things like code suggestions, parameter suggestions, code completion, the much loved ctrl + click — none of those would work. So while boring, it has to be done.
Oh and the mini-map has been re-implemented from scratch, so it’s now fast, accurate and responsive – and it works with mousewheel and keyboard.
Code suggestion
One of features that is bubbling up to the surface right now, is code suggestion. It’s something that we simply take for granted these days, and we dont really consider how much work it is to make. Eric has done a lot to simplify this for DWScript, but you still have to build up the codebase around it. But thankfully that is now largely done, leaving only a bit of styling and focus handling.
Form Files
In the previous build the IDE only recognized unit files (.pas), but in the current version it will check for an accompanying .DFM file. If a form-design file exists, it will open up a form-designer page rather than a pure code page.
The form-designer itself has received a bit of love lately too. Keyboard shortcuts have been added, such as holding down CTRL during a multiple selection — and changes to the layout is signaled back to the IDE and reflected in save-states changing (i.e if you change a form layout, the Save and Save All icon becomes enabled).
The form layout objects (visual widgets) have also been re-worked a bit. We want our DFM file-format to be JSON, so full JSON object persistence has been implemented. The form-designer widgets inherit from TQTXJSONPersistent, making it a literal one-liner to load and save form design.
We do need to wait for the AST explorer code to finish though, before you can start dragging & dropping widgets. Visual controls dont magically appear by themselves. Packages must be registered, and visual controls must likewise be registered with the IDE before they become known to the designer. So once the AST code is finished, we move on to packages – and finally glue the pieces together.
RTL advancements
The RTL has seen just as much changes as the IDE itself, and for good reason. Unlike “other” Web Technology tools, Quartex Pascal has an RTL that supports everything HTML5 has to offer. You are not limited to a static, fixed layout like we are used to under Delphi or Lazarus.
The ability to work with dynamic layout presents some interesting and highly efficient design opportunities. I find myself using the blocking layout model more and more, since it simplifies building up a dynamic UI that scales. Being able to work with different font scales too, like point (pt), as opposed to traditional pixel (px) closes the circle; it makes it possible to implement visual components that can scale it’s content to fit the container. This in turn simplifies writing software that renders well on both Desktop, Mobile and browser.
The changes has been too many to list here, but I have pretty much implemented all the event delegate objects (more to come), tweaked creation speed even further – and added additional polyfill files that ensures that your code works on every browser (a polyfill is a fallback system, so if a browser lacks a feature – the polyfill is used instead).
Application models
Under Quartex Pascal the TApplication object plays an important role, much more than you are used to under Delphi or Lazarus. It is TApplication that is responsible for maintaining your layout – and ultimately how forms are shown.
- If you are writing a mobile application you obviously want your forms to slide into view, just like native applications do on iPhone and Android.
- If you are writing a client-server website solution, you might prefer that forms covers the full width of the browser, with variable height – with the user switching forms by clicking on a toolbar, menu option or link.
- Perhaps you would like the forms to the stacked vertically, so that each form comes into view as the user scrolls downwards – perhaps with some fancy effect, or a static background behind the forms.
- And last but not least, you might prefer that your web application looks and behaves like a Windows desktop application. With multiple windows that can be moved around, a normal menu system on top of each window – or on top of the browser.
The only way to consolidate these diverse and even conflicting layout models, is to implement several TApplication classes; each one representing the layout model you want to work with. So when your create a project, you pick the layout model you want – and the correct TApplication is chosen and generated for your project.
Actual menus
The RTL have seen a few new widgets added, but the most interesting one is without a doubt the Menu widget. This is a widget that mimics how a normal menu works in a real program.
Creating a menu might not sound interesting, but it’s actually a small challenge under HTML. Not the coding itself, but dealing with menu presentation without visual artifacts. Whenever you click a menu that has a sub-menu attached, the new menu is created from code dynamically. It’s positioned at the end of it’s invoker (to the right of the parent menu item) and should only show up when all it’s child elements have been created.
This was very tricky to get right under a competing system, because the way elements was created was, well, wrong. You want to avoid reflows at all cost during the constructor – otherwise there will be visual artifacts and flickering. But that is not an issue under QTX. And the speed is insane. Even with 100 recursive items on a menu container, it’s virtually instantaneous.
If you are wondering why this makes any difference, you have to remember Quartex Media Desktop. This is not a simple toy with an onClick event, but can be bound into the process tree of the media desktop. The new code is barely 500 lines of code, the older version was over 3000 lines of code.
The goal for the IDE is that you can create a full desktop as a project. Not just programs that should run on the desktop (and its Ragnarok message protocol interface) – but the actual desktop system, which also covers several node.js system services.
The reason this is cool is because this enables you to deliver full scale, desktop level software purely through the browser. Such a desktop would be suitable for a school, a tutoring company, as an intranet – or for teams that need to share files, chat in realtime — and do their software development via the same web interface.
So it’s “a little bit” bigger than some mock desktop.
Come join the fun!
Want to support the project? All financial backers that donates $100+ get their name in the product, access to the full IDE source-code on completion, and access to the Quartex Media Desktop system (which is a complete web desktop with a clustered back-end, compiled to JavaScript and running on node.js. Portable, platform and chipset independent, and very powerful).
A smaller sum monthly is also welcome. The project would not exist without members backing it with $100, $200 etc every month. This both motivates and helps me allocate hours for continuous work.
When the IDE is finished you will also have access to the IDE source-code through special dispensation. Backers have rights since they have helped create the project.
All donations are welcome, both large and small. But donations over $100, especially reoccurring, is what drives this project forward.
Remember to send me a message on Facebook so I can add you to the Admin group: https://www.facebook.com/quartexnor/
Quartex Pascal, Build 10b is out for backers
I am deeply moved by some of the messages I have received about Quartex Pascal. Typically from people who either bought Smart Mobile studio or have followed my blog over the years. In short, developers that like to explore new ideas; people who also recognize some of the challenges large and complex run-time libraries like the VCL, FMX and LCL face in 2020.
Since I started with all this “compile to JavaScript” stuff, the world has changed. And while I’m not always right – I was right about this. JavaScript will and have evolved into a power-house. Largely thanks to Microsoft killing Basic. But writing large scale applications in JavaScript, that is extremely time consuming — which is where Quartex Pascal comes in.
Quartex Pascal evolves every weekend. There are at least 2 builds each weekend for backers. Why not become a backer and see the product come to life? Get instant access to new builds, the docs, and learn why QTX code runs so much faster than the alternatives?
A very important distinction
Let me first start by iterating what I mentioned in my previous post, namely that I am no longer involved with The Smart Company. Nor am I involved with Smart Mobile Studio. I realize that it can be difficult for some to separate me from that product, since I blogged and created momentum for it for more than a decade. But alas, life change and sometimes you just have to let go.
The QTX Framework which today has become a fully operational RTL was written by me back between 2013-2014 (when I was not working for the company due to my spinal injury). And the first version of that framework was released under an open-source license.
When I returned to The Smart Company, it was decided that to save time – we would pull the QTX Framework into the Smart RTL. Since I owned the QTX Framework, and it was open source, it was perfectly fine to include the code. The code was bound by the open source licensing model, so we broke no rules including it. And I gave dispensation that it could be included (although the original license explicitly stated that the units should remain untouched and separate, and only inherited from).
Quartex Media Desktop, a complete desktop system (akin to X Windows for Linux) written completely in Object Pascal, including a clustered, service oriented back-end. All of it done in Quartex Pascal — a huge project in its own right. Under Quartex Pascal, this is now a project type, which means you can have your own cloud solution at the click of a button.
As I left the company for good before joining Embarcadero, The Smart Company and myself came to an agreement that the parts of QTX that still exists in the Smart Mobile Studio RTL, could remain. It would be petty and small to make a huge number out of it, and I left on my own terms. No point ruining all that hard work we did. So we signed an agreement that underlined that there would be overlaps here and there in our respective codebases, and that the QTX Framework and Quartex Media Desktop was my property.
Minor overlaps
As mentioned there will be a few minor overlaps, but nothing substantial. The class hierarchy and architecture of the QTX RTL is so different, that 90% of the code in the Smart RTL simply won’t work. And I made it that way on purpose so there would be no debates over who made what.
QTX represents how I felt the RTL should have been done. And its very different from where Smart Mobile Studio ended up.
The overlaps are simple and few, but it can be helpful for Smart developers to know about if they plan on taking QTX for a test-drive:
- TInteger, TString and TVariant. These were actually ported from Delphi (part of the Sith Library, a pun on Delphi’s Jedi Library).
- TDataTypeConverter came in through the QTX Framework. It has been completely re-written from scratch. The QTX version is endian aware (works on both ARM, X86 and PPC). Classes that deal with binary data (like TStream, TBuffer etc) inherit from TDataTypeConverter. That way, you dont have to call a secondary instance just to perform conversion. This is easier and much more efficient.
- Low-level codecs likewise came from the QTX Framework, but I had to completely re-write the architecture. The old model could only handle binary data, while the new codec classes also covers text based formats. Codecs can be daisy-chained so you can do encoding, compression and encryption by feeding data into the first, and catching the processed data from the last codec in the chain. Very handy, especially when dealing with binary messages and database drivers.
- The in-memory dataset likewise came from the QTX Framework. This is probably the only unit that has remained largely unchanged. So that is a clear overlap between the Smart RTL and QTX.
- TextCraft is an open source library that covers Delphi, Freepascal and DWScript. The latter was pulled in and used as the primary text-parser in Smart. This is also the default parser for QTX, and have been largely re-written so it could be re-published under the Shareware license.
Since the QTX RTL is very different from Smart, I haven’t really bothered to use all of the old code. Stuff like the CSS Effects units likewise came from the QTX Framework, but the architecture I made for Smart is not compatible with QTX, so it makes no sense using that code. I ported my Delphi tweening library to DWScript in 2019, which was a part of my Patreon project. So most of the effects in QTX use our own tweening library. This has some very powerful perks, like being able to animate a property on any object rather than just a HTML Element. And you can use it for Canvas drawing too, which is nice.
Progress. Where are we now?
So, where am I in this work right now? The RTL took more or less 1 year to write from scratch. I only have the weekends for this type of work, and it would have been impossible without my backers. So I cannot thank each backer enough for the faith in this. The RTL and new IDE is actually just a stopping point on the road to a much bigger project, namely CloudForge, which is the full IDE running as an application on the Quartex Media Desktop. But first, let’s see what has been implemented!
AST unit view
The Unit Overview panel. Easy access to ancestor classes as links (still early R&D). And the entire RTL on a second tab. This makes it very easy to learn the new RTL. There is also proper documentation, both as PDF and standard helpfile.
When the object-pascal code is compiled by DWScript, it goes through a vigorous process of syntax checking, parsing, tokenizing and symbolization (or objectification is perhaps a better word), where every inch of the code is transformed into objects that the compiler can work with and produce code from. These symbols are isolated in what is known as an AST, short for “Abstract Symbol Tree”. Basically a massive in-memory tree structure that contains your entire program reduced to symbols and expressions.
In order for us to have a live structural view of the current unit, I have created a simple background process that compiles the current unit, grabs the resulting AST, locates the unit symbol, and then displays the information in a meaningful way. This is the exact same as most other IDE’s do, be it Visual Studio, Embarcadero Delphi, or Lazarus.
So we have that in place already. I also want to make it more elaborate, where you can click yourself to glory by examining ancestors, interfaces, partial class groups – as well as an option to include inherited members (which should be visually different). Either way, the AST code is done. Just need to consolidate a few tidbits so that each Treeview node retains information about source-code location (so that when you double-click a symbol, the IDE navigates to where the symbol exists in the codebase).
JavaScript parsing and compilation
QTX doesn’t include just one compiler, but three. In order for the unit structure to also work for JavaScript files I have modified Besen, which is an ES5 compatible JavaScript engine written in Delphi. And I use this much like DWScript to parse and work with the AST.
Besen is a wonderful compiler. Where DWScript produces JavaScript from Object Pascal, Besen produces bytecodes from JavaScript (which are further JIT compiled). This opens up for some interesting options. I need to add support for ES6 though, modules and require are especially important for modern node.js programming (and yes, the QTX RTL supports these concepts)
HTML5 Rendering and CSS preview
Instead of using Chromium inside the IDE, which is pretty demanding, I have decided to go for HTMLComponents to deal with “normal” tasks. The “Welcome” tab-page for example — it would be complete overkill to use a full Chromium instance just for that, and TEdgeBrowser is likewise shooting sparrows with a Bazooka.
THTMLComponents have a blistering fast panel control that can render more or less any HTML5 document you throw at it (much better than the old TFrameViewer component). But obviously, it doesn’t have JS support. But we won’t be using JS when displaying static information – or indeed, editing HTML5 compliant content.
WYSIWYG Editor
The biggest benefit for HTMLComponents, is that it’s a fully operational HTML compliant editor. Which means you can do more or less all your manual design with that editor. In Quartex Pascal there is direct support for HTML files. Quartex works much like Visual Studio code, except it has visual designers. So you can create a HTML file and either type in the code manually, or switch to the HTMLComponents editor.
Which is what products like Help & Manual uses it for
Image from HTMLComponents application gallery website
Support for HTML, CSS and JS files directly
While not new, this is pretty awesome. Especially since we can do a bit of AST navigation here too to present similar information as we do for Object Pascal. The whole concept behind the QTX RTL, is that you have full control over everything. You can stick to a normal Delphi like form designer and absolute positioning, or you can opt for a more dynamic approach where you create content via code. This is perfect for modern websites that blend scrolling, effects and content (both dynamic and static content) for a better user experience.
You can even (spoiler alert), take a piece of HTML and convert it into visual controls at runtime. That is a very powerful function, because when doing large-scale, elaborate custom controls – you can just tell the RTL “hey, turn this piece of HTML into a visual control for me, and deliver it back when you are ready).
Proper Form Designer
Writing a proper form designer like Delphi has is no walk in the park. It has to deal not just with a selected control, but also child elements. It also has to be able to select multiple elements based on key-presses (shift + click adds another item to the selection), or from the selection rectangle.
A property form layout control. Real-time rendering of controls is also possible, courtesy of HTMLComponents. But to be honest, it just gets in the way. Its much easier to work with this type of designer. It’s fast, responsive, accurate and will have animated features that makes it a joy to work with.Â
Well, that’s not going to be a problem. I spent a considerable amount of time writing a proper form designer, one that takes both fixed and dynamic content into account. So the Quartex form designer handles both absolute and stacked layout modes (stacked means top-down, what in HTML is knock as blocking element display, where each section stretch to the full width, and only have a defined height [that you can change]).
Node.js Service Protocol Designer
Writing large-scale servers, especially clustered ones, is very fiddly in vanilla JavaScript under node.js. It takes 3 seconds to create a server object, but as we all know, without proper error handling, a concurrent message format, modern security and a solid framework to handle it all — that “3 second” thing falls to the ground quickly.
This is where the Ragnarok message system comes in. Which is both a message framework, and a set of custom servers adapted for dealing with that type of data. It presently supports WebSocket, TCP and UDP. But expanding that to include REST is very easy.
This is where the full might of the QTX Framework begins to dawn. As i wrote before we started on the Quartex Media Desktop (Which this IDE and RTL is a part of), in the future developers wont just drag & drop components on a form; they will drag & drop entire ecosystems ..
But the power of the system is not just in how it works, and how you can create your own protocols, and then have separate back-end services deal with one part of your infrastructure’s workload. It is because you can visually design the protocols using the Node Builder. This is being moved into the QTX IDE as I type. So should make it for Build 12 next weekend.
In short, you design your protocols, messages and types – a bit like RemObjects SDK if you have used that. And the IDE generates both server and client code for you. All you have to do is fill in the content that acts on the messages. Everything else is handled by the server components.
Suddenly, you can spend a week writing a large-scale, platform agnostic service stack — that would have taken JavaScript developers months to complete. And instead of having to manage a 200.000 lines codebase in JavaScript — you can enjoy a 4000 line, easily maintainable Object Pascal codebase.
Build 11
Im hoping to have build 11 out tomorrow (Sunday) for my backers. Im still experimenting a bit with the symbol information panel, since I want it to be informative not just for classes, but also for methods and properties. Making it easy to access ancestor implementations etc.
I also need to work a bit more on the JS parsing. Under ES5 its typical to use variables to hold objects (which is close to how we think of a class), so composite and complex datatypes must be expanded. I also need to get symbol position to work property, because since Besen is a proper bytecode compiler, it doesn’t keep as much information in it’s AST as DWScript does.
Widgets (which is what visual controls are called under QTX) should appear in build 12 or 13. The IDE supports zip-packages. The file-source system I made for the TVector library (published via Embarcadero’s website a few months back) allows us to mount not just folders as a file-source, but also zip files. So QTX component packages will be ordinary zip-files containing the .pas files, asset files and a metadata descriptor file that tells the IDE what to expect. Simple, easy and very effective.
Support the project!
Want to support the project? All financial backers that donates $100+ get their name in the product, access to the full IDE source-code on completion, and access to the Quartex Media Desktop system (which is a complete web desktop with a clustered back-end, compiled to JavaScript and running on node.js. Portable, platform and chipset independent, and very powerful).
Your help matters! It pays for components, hours and above all, tools and motivation. In return, you get full access to everything and a perpetual license. No backers will ever pay a cent for any future version of Quartex Pascal. Note: PM me after donating so I can get you added to the admin group! Click here to visit paypal: https://www.paypal.com/paypalme/quartexNOR
All donations are welcome, both large and small. But donations over $100, especially reoccurring, is what drives this project forward. It also gets you access to the Quartex Developer group on Facebook where new builds, features etc is happening. It is the best way to immediately get the latest build, read documentation as its written and see the product come to life!
Nodebuilder, QTX and the release of my brand new social platform
First, let me wish everyone a wonderful new year! With xmas and the silly season firmly behind us, and my batteries recharged – I feel my coding fingers itch to get started again.
2019 was a very busy year for me. Exhausting even. I have juggled both a full time job, kids and family, as well as our community project, Quartex Media Desktop. And since that project has grown considerably, I had to stop and work on the tooling. Which is what NodeBuilder is all about.
I have also released my own social media platform (see further down). This was initially scheduled for Q4 2020, but Facebook pissed me off something insanely, so I set it up in december instead.
NodeBuilder
For those of you that read my blog you probably remember the message system I made for the Quartex Desktop, called Ragnarok? This is a system for dealing with message dispatching and RPC, except that the handling is decoupled from the transport medium. In other words, it doesnt care how you deliver a message (WebSocket, UDP, REST), but rather takes care of serialization, binary data and security.
All the back-end services that make up the desktop system, are constructed around Ragnarok. So each service exposes a set of methods that the core can call, much like a normal RPC / SOAP service would. Each method is represented by a request and response object, which Ragnarok serialize to a JSON message envelope.
In our current model I use WebSocket, which is a full duplex, long-term connection (to avoid overhead of having to connect and perform a handshake for each call). But there is nothing in the way of implementing a REST transport layer (UDP is already supported, it’s used by the Zero-Config system. The services automatically find each other and register, as long as they are connected to the same router or switch). For the public service I think REST makes more sense, since it will better utilize the software clustering that node.js offers.
Node Builder is a relatively simple service designer, but highly effective for our needs
Now for small services that expose just a handful of methods (like our chat service), writing the message classes manually is not really a problem. But the moment you start having 20 or 30 methods – and need to implement up to 60 message classes manually – this approach quickly becomes unmanageable and prone to errors. So I simply had to stop before xmas and work on our service designer. That way we can generate the boilerplate code in seconds rather than days and weeks.
While I dont have time to evolve this software beyond that of a simple service designer (well, I kinda did already), I have no problem seeing this system as a beginning of a wonderful, universal service authoring system. One that includes coding, libraries and the full scope of the QTX runtime-library.
In fact, most of the needed parts are in the codebase already, but not everything has been activated. I don’t have time to build both a native development system AND the development system for the desktop.
NodeBuilder already have a fully functional form designer and code editor, but it is dormant for now due to time restrictions. Quartex Media Desktop comes first
But right now, we have bigger fish to fry.
Quartex Media Desktop
We have made tremendous progress on our universal desktop environment, to the point where the baseline services are very close to completion. A month should be enough to finish this unless something unforeseen comes up.
Quartex Media Desktop provides an ecosystem for advanced web applications
You have to factor in that, this project has only had weekends and the odd after work hours allocated for it. So even though we have been developing this for 12 months, the actual amount of days is roughly half of that.
So all things considered I think we have done a massive amount of code in such a short time. Even simple 2d games usually take 2 years of daily development, and that includes a team of at least 5 people! Im a single developer working in my spare time.
So what exactly is left?
The last thing we did before xmas was upon us, was to throw out the last remnants of Smart Mobile Studio code. The back-end services are now completely implemented in our own QTX runtime-library, which has been written from scratch. There is not a line of code from Smart Mobile Studio in QTX, which means we no longer have to care what that system does or where it goes.
To sum up:
- Push all file handling code out of the core
- Implement file-handling as it’s own service
Those two steps might seem simple enough, but you have to remember that the older code was based on the Linux path system, and was read-only.
So when pushing that code out of the core, we also have to add all the functionality that was never implemented in our prototype.
Each class actually represents a separate “mini” program, and there are still many more methods to go before we can put this service into production.
Since Javascript does not support threads, each method needs to be implemented as a separate program. So when a method is called, the file/task manager literally spawns a new process just for that task. And the result is swiftly returned back to the caller in async manner.
So what is ultimately simple, becomes more elaborate if you want to do it right. This is the price we pay for universality and a cluster enabled service-stack.
This is also why I have put the service development on pause until we have finished the NodeBuilder tooling. And I did this because I know by experience that the moment the baseline is ready, both myself and users of the system is going to go “oh we need this, and that and those”. Being able to quickly design and auto-generate all the boilerplate code will save us months of work. So I would rather spend a couple of weeks on NodeBuilder than wasting months having to manually write all that boilerplate code down the line.
What about the QTX runtime-library?
Writing an RTL from scratch was not something I could have anticipated before we started this project. But thankfully the worst part of this job is already finished.
The RTL is divided into two parts:
Non Visual code. Classes and methods that makes QTX an RTL- Visual code. Custom Controls + standard controls (buttons, lists etc)
Visual designer
As you can see, the non-visual aspect of the system is finished and working beautifully. It’s a lot faster than the code I wrote for Smart Mobile Studio (roughly twice as fast on average). I also implemented a full visual designer, both as a Delphi visual component and QTX visual component.
Quartex Media Desktop makes running on several machines [cluster] easy and seamless
Why is the QTX Runtime-Library important again?
When the desktop is out the door, the true work begins! The desktop has several roles to play, but the most important aspect of the desktop – is to provide an ecosystem capable of hosting web based applications. Offering features and methods traditionally only found in Windows, Linux or OS X. It truly is a complete cloud system that can scale from a single affordable SBC (single board computer), to a high-end cluster of powerful servers.
Clustering and writing distributed applications has always been difficult, but Quartex Media Desktop makes it simple. It is no more difficult for a user to work on a clustered system, as it is to work on a normal, single OS. The difficult part has already been taken care of, and as long as people follow the rules, there will be no issues beyond ordinary maintenance.
And the first commercial application to come out of Quartex Components, is Cloud Forge, which is the development system for the platform. It has the same role as Visual Studio for Windows, or X Code for Apple OS X.
The Quartex Media Desktop Cluster cube. A $400 super computer
I have prepared 3 compilers for the system already. First there is C/C++ courtesy of Clang. So C developers will be able to jump in and get productive immediately. The second compiler is freepascal, or more precise pas2js, which allows you to compile ordinary freepascal code (which is highly Delphi compatible) to both JavaScript and WebAssembly.
And last but not least, there is my fork of DWScript, which is the same compiler that Smart Mobile Studio uses. Except that my fork is based on the absolute latest version, and i have modified it heavily to better match special features in QTX. So right out of the door CloudForge will have C/C++, two Object Pascal compilers, and vanilla Javascript and typescript. TypeScript also has its own WebAssembly compiler, so doing hard-core development directly in a browser or HTML5 viewport is where we are headed.
Once the IDE is finished I can finally, finally continue on the LDEF bytecode runtime, which will be used in my BlitzBasic port and ultimately replace both clang, freepascal and DWScript. As a bonus it will emit native code for a variety of systems, including x86, ARM, 68k [including 68080] and PPC.
This might sound incredibly ambitious, if not impossible. But what I’m ultimately doing here -is moving existing code that I already have into a new paradigm.
The beauty of object pascal is the sheer size and volume of available components and code. Some refactoring must be done due to the async nature of JS, but when needed we fall back on WebAssembly via Freepascal (WASM executes linear, just like ordinary native code does).
A brand new social platform
During december Facebook royally pissed me off. I cannot underline enough how much i loath A.I censorship, and the mistakes that A.I does – in which you are utterly powerless to complain or be heard by a human being. In my case i posted a gif from their own mobile application, of a female body builder that did push-ups while doing hand-stands. In other words, a completely harmless gif with strength as the punchline. The A.I was not able to distinguish between a leotard and bare-skin, and just like that i was muted for over a week. No human being would make such a ruling. As an admin of a fairly large set of groups, there are many cases where bans are the result. Disgruntled members that acts out of revenge and report technical posts about coding as porn or offensive. Again, you are helpless because there are nobody you can talk to about resolving the issue. And this time I had enough.
It was always planned that we would launch our own social media platform, an alternative to Facebook aimed at adult geeks rather than kids (Facebook operates with an age limit of 12 years). So instead of waiting I rushed out and set up a brand new social network. One where those banale restrictions Facebook has conditioned us with, does not apply.
Just to underline, this is not some simple and small web forum. This is more or less a carbon copy of Facebook the way it used to be 8-9 years ago. So instead of having a single group on facebook, we can now have as many groups as we like, on a platform that looks more or less identical to Facebook – but under our control and human rules.
Amigadisrupt.com is a brand new social media platform for geeks
You can visit the site right now at https://www.amigadisrupt.com. Obviously the major content on the platform right now is dominated by retro computing – but groups like Delphi Developer and FPC developer has already been setup and are in use. But if you are expecting thousands of active users, that will take time. We are now closing in on 250 active users which is pretty good for such a short period of time. I dont want a platform anywhere near as big as FB. The goal is to get 10k users and have a stable community of coders, retro geeks, builders and creative individuals.
AD (Amiga Disrupt) will be a standard application that comes with Quartex Media Desktop. This is the beauty of web technology, in that it can unify different resources under one roof. And we will have our cake and eat it come hell or high water.
Disclaimer: Amiga Disrupt has a lower age limit of 18 years. This is a platform meant for adults. Which means there will be profanity, jokes that would get you banned on Facebook and content that is not meant for kids. This is hacker-land, and political correctness is considered toilet paper. So if you need social toffery like FB and Twitter deals with, you will be kicked by one of the admins.
After you sign up your feed will be completely empty. Here is how to get it started. And feel free to add me to your friends-list!
Quartex “Cloud Ripper” hardware
For close to a year now I have been busy on a very exciting project, namely my own cloud system. While I have written about this project quite a bit these past months, mostly focusing on the software aspect, not much has been said about that hardware.
Quartex “Cloud Ripper” running neatly on my home-office desk
So let’s have a look at Cloud Ripper, the official hardware setup for Quartex Media Desktop.
Tiny footprint, maximum power
Despite its complexity, the Quartex Media Desktop architecture is surprisingly lightweight. The services that makes up the baseline system (read: essential services) barely consume 40 megabytes of ram per instance (!). And while there is a lot of activity going on between these services -most of that activity is message-dispatching. Sending messages costs practically nothing in cpu and network terms. This will naturally change the moment you run your cloud as a public service, or setup the system in an office environment for a team. The more users, the more signals are shipped between the processes – but with the exception of reading and writing large files, messages are delivered practically instantaneous and hardly use CPU time.
Quartex Media Desktop is based on a clustered micro-service architecture
One of the reasons I compile my code to JavaScript (Quartex Media Desktop is written from the ground up in Object Pascal, which is compiled to JavaScript) has to do with the speed and universality of node.js services. As you might know, Node.js is powered by the Google V8 runtime engine, which means the code is first converted to bytecodes, and further compiled into highly optimized machine-code [courtesy of llvm]. When coded right, such Javascript based services execute just as fast as those implemented in a native language. There simply are no perks to be gained from using a native language for this type of work. There are however plenty of perks from using Node.js as a service-host:
- Node.js delivers the exact same behavior no matter what hardware or operating-system you are booting up from. In our case we use a minimal Linux setup with just enough infrastructure to run our services. But you can use any OS that supports Node.js. I actually have it installed on my Android based Smart-TV (!)
- We can literally copy our services between different machines and operating systems without recompiling a line of code. So we don’t need to maintain several versions of the same software for different systems.
- We can generate scripts “on the fly”, physically ship the code over the network, and execute it on any of the machines in our cluster. While possible to do with native code, it’s not very practical and would raise some major security concerns.
- Node.js supports WebAssembly, you can use the Elements Compiler from RemObjects to write service modules that executes blazingly fast yet remain platform and chipset independent.
The Cloud-Ripper cube
The principal design goal when I started the project, was that it should be a distributed system. This means that instead of having one large-service that does everything (read: a typical “native” monolithic design), we instead operate with a microservice cluster design. Services that run on separate SBC’s (single board computers). The idea here is to spread the payload over multiple mico-computers that combined becomes more than the sum of their parts.
Cloud Ripper – Based on the Pico 5H case and fitted with 5 x ODroid XU4 SBC’s
So instead of buying a single, dedicated x86 PC to host Quartex Media Desktop, you can instead buy cheap, off-the-shelves, easily available single-board computers and daisy chain them together. So instead of spending $800 (just to pin a number) on x86 hardware, you can pick up $400 worth of cheap ARM boards and get better network throughput and identical processing power (*). In fact, since Node.js is universal you can mix and match between x86, ARM, Mips and PPC as you see fit. Got an older PPC Mac-Mini collecting dust? Install Linux on it and get a few extra years out of these old gems.
(*) A single XU4 is hopelessly underpowered compared to an Intel i5 or i7 based PC. But in a cluster design there are more factors than just raw computational power. Each board has 8 CPU cores, bringing the total number of cores to 40. You also get 5 ARM Mali-T628 MP6 GPUs running at 533MHz. Only one of these will be used to render the HTML5 display, leaving 4 GPUs available for video processing, machine learning or compute tasks. Obviously these GPUs won’t hold a candle to even a mid-range graphics card, but the fact that we can use these chips for audio, video and computation tasks makes the system incredibly versatile.
Another design goal was to implement a UDP based Zero-Configuration mechanism. This means that the services will find and register with the core (read: master service) automatically, providing the machines are all connected to the same router or switch.
Put together your own supercomputer for less than $500
The first “official” hardware setup is a cluster based on 5 cheap ARM boards; namely the ODroid XU4. The entire setup fits inside a Pico Cube, which is a special case designed to house this particular model of single board computers. Pico offers several different designs, ranging from 3 boards to a 20 board super-cluster. You are not limited ODroid XU4 boards if you prefer something else. I picked the XU4 boards because they represent the lowest possible specs you can run the Quartex Media Desktop on. While the services themselves require very little, the master board (the board that runs the QTXCore.js service) is also in charge of rendering the HTML5 display. And having tested a plethora of boards, the ODroid XU4 was the only model that could render the desktop properly (at that low a price range).
Note: If you are thinking about using a Raspberry PI 3B (or older) as the master SBC, you can pretty much forget it. The media desktop is a piece of very complex HTML5, and anything below an ODroid XU4 will only give you a terrible experience (!). You can use smaller boards as slaves, meaning that they can host one of the services, but the master should preferably be an ODroid XU4 or better. The ODroid N2 [with 4Gb Ram] is a much better candidate than a Raspberry PI v4. A Jetson Nano is an even better option due to its extremely powerful GPU.
Booting into the desktop
One of the things that confuse people when they read about the desktop project, is how it’s possible to boot into the desktop itself and use Quartex Media Desktop as a ChromeOS alternative?
How can a “cloud platform” be used as a desktop alternative? Don’t you need access to the internet at all times? If it’s a server based system, how then can we boot into it? Don’t we need a second PC with a browser to show the desktop?
Accessing the desktop like a “web-page” from a normal Linux setup
To make a long story short: the “master” in our cluster architecture (read: the single-board computer defined as the boss) is setup to boot into a Chrome browser display under “kiosk mode”. When you start Chrome in kiosk mode, this removes all traces of the ordinary browser experience. There will be no toolbars, no URL field, no keyboard shortcuts, no right-click popup menus etc. It simply starts in full-screen and whatever HTML5 you load, has complete control over the display.
What I have done, is to to setup a minimal Linux boot sequence. It contains just enough Linux to run Chrome. So it has all the drivers etc. for the device, but instead of starting the ordinary Linux Desktop (X or Wayland) -we instead start Chrome in kiosk mode.
Booting into the same desktop through Chrome in Kiosk Mode. In this mode, no Linux desktop is required. The Linux boot sequence is altered to jump straight into Chrome
Chrome is started to load from 127.0.0.1 (this is a special address that always means “this machine”), which is where our QTXCore.js service resides that has it’s own HTTP/S and Websocket servers. The client (HTML5 part) is loaded in under a second from the core — and the experience is more or less identical to starting your ChromeBook or NAS box. Most modern NAS (network active storage) devices are much more than a file-server today. NAS boxes like those from Asustor Inc have HDMI out, ships with a remote control, and are designed to act as a media center. So you connect the NAS directly to your TV, and can watch movies and listen to music without any manual conversion etc.
In short, you can setup Quartex Media Desktop to do the exact same thing as ChromeOS does, booting straight into the web based desktop environment. The same desktop environment that is available over the network. So you are not limited to visiting your Cloud-Ripper machine via a browser from another computer; nor are you limited to just using a dedicated machine. You can setup the system as you see fit.
Why should I assemble a Cloud-Ripper?
Getting a Cloud-Ripper is not forced on anyone. You can put together whatever spare hardware you have (or just run it locally under Windows). Since the services are extremely lightweight, any x86 PC will do. If you invest in a ODroid N2 board ($80 range) then you can install all the services on that if you like. So if you have no interest in clustering or building your own supercomputer, then any PC, Laptop or IOT single-board computer(s) will do. Provided it yields more or equal power as the XU4 (!)
What you will experience with a dedicated cluster, regardless of putting the boards in a nice cube, is that you get excellent performance for very little money. It is quite amazing what $200 can buy you in 2019. And when you daisy chain 5 ODroid XU4 boards together on a switch, those 5 cheap boards will deliver the same serving power as an x86 setup costing twice as much.
The NVidia Jetson Nano SBC, one of the fastest boards available at under $100
Pico is offering 3 different packages. The most expensive option is the pre-assembled cube. This is for some reason priced at $750 which is completely absurd. If you can operate a screwdriver, then you can assemble the cube yourself in less than an hour. So the starter-kit case which costs $259 is more than enough.
Next, you can buy the XU4 boards directly from Hardkernel for $40 a piece, which will set you back $200. If you order the Pico 5H case as a kit, that brings the sub-total up to $459. But that price-tag includes everything you need except sd-cards. So the kit contains power-supply, the electrical wiring, a fast gigabit ethernet switch [built-into the cube], active cooling, network cables and power cables. You don’t need more than 8Gb sd-cards, which costs practically nothing these days.
Note: The Quartex Media Desktop “file-service” should have a dedicated disk. I bought a 256Gb SSD disk with a USB 3.0 interface, but you can just use a vanilla USB stick to store user-account data + user files.
As a bonus, such a setup is easy to recycle should you want to do something else later. Perhaps you want to learn more about Kubernetes? What about a docker-swarm? A freepascal build-server perhaps? Why not install FreeNas, Plex, and a good backup solution? You can set this up as you can afford. If 5 x ODroid XU4 is too much, then get 3 of them instead + the Pico 3H case.
So should Quartex Media Desktop not be for you, or you want to do something else entirely — then having 5 ODroid XU4 boards around the house is not a bad thing.
Oh and if you want some serious firepower, then order the Pico 5H kit for the NVidia Jetson Nano boards. Graphically those boards are beyond any other SoC on the market (in it’s price range). But as a consequence the Jetson Nano starts at $99. So for a full kit you will end up with $500 for the boards alone. But man those are the proverbial Ferrari of IOT.
Quartex Media Desktop, new compiler and general progress
It’s been a few weeks since my last update on the project. The reason I dont blog that often about Quartex Media Desktop (QTXMD), is because the official user-group has grown to 2000+ members. So it’s easier for me to post developer updates directly to the audience rather than writing articles about it.
Quartex Media Desktop ~ a complete environment that runs on every device
If you haven’t bothered digging into the project, let me try to sum it up for you quickly.
Quick recap on Quartex Media Desktop
To understand what makes this project special, first consider the relationship between Microsoft Windows and a desktop program. The operating system, be it Windows, Linux or OSX – provides an infrastructure that makes complex applications possible. The operating-system offers functions and services that programs can rely on.
The most obvious being:
- A filesystem and the ability to save and load data
- A windowing toolkit so programs can be displayed and have a UI
- A message system so programs can communicate with the OS
- A service stack that takes care of background tasks
- Authorization and identity management (security)
I have just described what the Quartex Media Desktop is all about. The goal is simple:
to provide for JavaScript what Windows and OS X provides for ordinary programs.
Just stop and think about this. Every “web application” you have ever seen, have all lacked these fundamental features. Sure you have libraries that gives you a windowing environment for Javascript, like Embarcadero Sencha; but im talking about something a bit more elaborate. Creating windows and buttons is easy, but what about ownership? A runtime environment has to keep track of the resources a program allocates, and make sure that security applies at every step.
Target audience and purpose
Take a second and think about how many services you use that have a web interface. In your house you probably have a router, and all routers can be administered via the browser. Sadly, most routers operate with a crude design and that leaves much to be desired.
Router interfaces for web are typically very limited and plain looking. Imagine what NetGear could do with Quartex Media Desktop instead
If you like to watch movies you probably have a Plex or Kodi system running somewhere in your house; perhaps you access that directly via your TV – or via a modern media system like Playstation 4 or XBox one. Both Plex and Kodi have web-based interfaces.
Netflix is now omnipresent and have practically become an institution in it’s own right. Netflix is often installed as an app – but the app is just a thin wrapper around a web-interface. That way they dont have to code apps for every possible device and OS out there.
If you commute via train in Scandinavia, chances are you buy tickets on a kiosk booth. Most of these booths run embedded software and the interface is again web based. That way they can update the whole interface without manually installing new software on each device.
Plex is a much loved system. It is based on a mix of web and native technologies
These are just examples of web based interfaces you might know and use; devices that leverage web technology. As a developer, wouldn’t it be cool if there was a system that could be forked, adapted and provide advanced functionality out of the box?
Just imagine a cheap Jensen router with a Quartex Media Desktop interface! It could provide a proper UI interface with applications that run in a windowing environment. They could disable ordinary desktop functionality and run their single application in kiosk mode. Taking full advantage of the underlying functionality without loss of security.
And the same is true for you. If you have a great idea for a web based application, you can fork the system, adjust it to suit your needs – and deploy a cutting edge cloud system in days rather than months!
New compiler?
Up until recently I used Smart Mobile Studio. But since I have left that company, the matter became somewhat pressing. I mean, QTXMD is an open-source system and cant really rely on third-party intellectual property. Eventually I fired up Delphi, forked the latest DWScript, and used that to roll a new command-line compiler.
Web technology has reached a level of performance that rivals native applications. You can pretty much retire Photoshop in favour of web based applications these days
But with a new compiler I also need a new RTL. Thankfully I have been coding away on the new RTL for over a year, but there is still a lot of work to do. I essentially have to implement the same functionality from scratch.
There will be more info on the new compiler / codegen when its production ready.
Progress
If I was to list all the work I have done since my last post, this article would be a small book. But to sum up the good stuff:
- Authentication has been moved into it’s own service
- The core (the main server) now delegates login messages to said service
- We no longer rely on the Smart Pascal filesystem drivers, but use the raw node.js functions instead (faster)
- The desktop now use the Smart Theme engine. This means that we can style the desktop to whatever we like. The OS4 theme that was hardcoded will be moved into its own proper theme-file. This means the user can select between OS4, iOS, Android and Ubuntu styling. Creating your own theme-files is also possible. The Smart theme-engine will be replaced by a more elaborate system in QTX later
- Ragnarok (the message api) messages now supports routing. If a routing structure is provided, the core will relay the message to the process in question (providing security allows said routing for the user)
- The desktop now checks for .info files when listing a directory. If a file is accompanied by an .info file, the icon is extracted and shown for that file
- Most of the service layer now relies on the QTX RTL files. We still have some dependencies on the Smart Pascal RTL, but we are making good progress on QTX. Eventually the whole system will have no dependencies outside QTX – and can thus be compiled without any financial obligations.
- QTX has it’s own node.js classes, including server and client base-classes
- Http(s) client and server classes are added to QTX
- Websocket and WebSocket-Secure are added to QTX
- TQTXHybridServer unifies http and websocket. Meaning that this server type can handle both orinary http requests – but also websocket connections on the same network socket. This is highly efficient for websocket based services
- UDP classes for node.js are implemented, both client and server
- Zero-Config classes are now added. This is used by the core for service discovery. Meaning that the child services hosted on another machine will automatically locate the core without knowing the IP. This is very important for machine clustering (optional, you can define a clear IP in the core preferences file)
- Fixed a bug where the scrollbars would corrupt widget states
- Added API functions for setting the scrollbars from hosted applications (so applications can tell the desktop that it needs scrollbar, and set the values)
- .. and much, much more
I will keep you all posted about the progress — the core (the fundamental system) is set for release in december – so time is of the essence! Im allocating more or less all my free time to this, and it will be ready to rock around xmas.
When the core is out, I can focus solely on the applications. Everything from Notepad to Calculator needs to be there, and more importantly — the developer tools. The CloudForge IDE for developers is set for 2020. With that in place you can write applications for iOS, Android, Windows, OS X and Linux directly from Quartex Media Desktop. Nothing to install, you just need a modern browser and a QTX account.
The system is brilliant for small teams and companies. They can setup their own instance, communicate directly via the server (text chat and video chat is scheduled) and work on their products in concert.
Calling node.js from Delphi
We got a good question about how to start a node.js program from Delphi on our Facebook group today (third one in a week?). When you have been coding for years you often forget that things like this might not be immediately obvious. Hopefully I can shed some light on the options in this post.
Node or chrome?
Just to be clear: node.js has nothing to do with chrome or chromium embedded. Chrome is a web-browser, a completely visual environment and ecosystem.
Node.js is the complete opposite. It is purely a shell based environment, meaning that it’s designed to run services and servers, with emphasis on the latter.
The only thing node.js and chrome have in common, is that they both use the V8 JavaScript runtime engine to load, JIT compile and execute scripts at high speed. Beyond that, they are utterly alien to each other.
Can node.js be embedded into a Delphi program?
Technically there is nothing stopping a C/C++ developer from compiling the node.js core system as C++ builder compatible .obj files; files that can then be linked into a Delphi application through references. But this also requires a bit of scaffolding, like adding support for malloc_, free_ and a few other procedures – so that your .obj files uses the same memory manager as your Delphi code. But until someone does just that and publish it, im afraid you are stuck with two options:
- Use a library called Toby, that keeps node.js in a single DLL file. This is the most practical way if you insist on hosting your own version of node.js
- Add node.js as a prerequisite and give users the option to locate the node.exe in your application’s preferences. This is the way I would go, because you really don’t want to force users to stick with your potentially outdated or buggy build.
So yes, you can use toby and just add the toby dll file to your program folder, but I have to strongly advice against that. There is no point setting yourself up for maintaining a whole separate programming language, just because you want JavaScript support.
“How many in your company can write high quality WebAssembly modules?”
If all you want to do is support JavaScript in your application, then I would much rather install Besen into Delphi. Besen is a JavaScript runtime engine written in Freepascal. It is fully compatible with Delphi, and follows the ECMA standard to the letter. So it is extremely compatible, fast and easy to use.
Like all Delphi components Besen is compiled into your application, so you have no dependencies to worry about.
Starting a node.js script
The easiest way to start a node.js script, is to simply shell-execute out of your Delphi application. This can be done as easily as:
ShellExecute(Handle, 'open', PChar('node.exe'), pchar('script.js'), nil, SW_SHOW);
This is more than enough if you just want to start a service, server or do some work that doesn’t require that you capture the result.
If you need to capture the result, the data that your node.js program emits on stdout, there is a nice component in the Jedi Component Library. Also plenty of examples online on how to do that.
If you need even further communication, you need to look for a shell-execute that support pipes. All node.js programs have something called a message-channel in the Javascript world. In reality though, this is just a named pipe that is automatically created when your script starts (with the same moniker as the PID [process identifier]).
If you opt for the latter you have a direct, full duplex message channel directly into your node.js application. You just have to agree with yourself on a protocol so that your Delphi code understands what node.js is saying, and visa versa.
UDP or TCP
If you don’t want to get your hands dirty with named pipes and rolling your own protocol, you can just use UDP to let your Delphi application communicate with your node.js process. UDP is practically without cost since its fundamental to all networking stacks, and in your case you will be shipping messages purely between processes on localhost. Meaning: packets are never sent on the network, but rather delegated between processes on the same machine.
In that case, I suggest you ship in the port you want your UDP server to listen on, so that your node.js service acts as the server. A simple command-line statement like:
node.exe myservice.js 8090
Inside node.js you can setup an UDP server with very little fuzz:
function setupServer(port) {
var os = require("os");
var dgram = require("dgram");
var socket = dgram.createSocket("udp4");
var MULTICAST_HOST = "224.0.0.236";
var BROADCAST_HOST = "255.255.255.255";
var ALL_PORT = 60540;
var MULTICAST_TTL = 1; // Local network
socket.bind(port);
socket.on('listening', function() {
socket.setMulticastLoopback(true);
socket.setMulticastTTL(MULTICAST_TTL);
socket.addMembership(multicastHost);
if(broadcast) { socket.setBroadcast(true); }
});
socket.on('message', parseMessage);
}
function parseMessage(message, rinfo) {
try {
var messageObject = JSON.parse(message);
var eventType = messageObject.eventType;
} catch(e) {
}
}
Note: the code above assumes a JSON text message.
You can then use any Delphi UDP client to communicate with your node.js server, Indy is good, Synapse is a good library with less overhead – there are many options here.
Do I have to learn Javascript to use node.js?
If you download DWScript you can hook-up the JS-codegen library (see library folder in the DWScript repository), and use that to compile DWScript (object pascal) to kick-ass Javascript. This is the same compiler that was used in Smart Mobile Studio.
“Adding WebAssembly to your resume is going to be a hell of a lot more valuable in the years to come than C# or Java”
Another alternative is to use Freepascal, they have a pas2js project where you can compile ordinary object-pascal to javascript. Naturally there are a few things to keep in mind, both for DWScript and Freepascal – like avoiding pointers. But clean object pascal compiles just fine.
If JavaScript is not your cup of tea, or you simply don’t have time to learn the delicate nuances between the DOM (document object model, used by browsers) and the 100% package oriented approach deployed by node.js — then you can just straight up to webassembly.
RemObjects Software has a kick-ass webassembly compiler, perfect if you dont have the energy or time to learn JavaScript. As of writing this is the fastest and most powerful toolchain available. And I have tested them all.
WebAssembly, no Javascript needed
You might remember Oxygene? It used to be shipped with Delphi as a way to target Microsoft CLR (common language runtime) and the .net framework.
Since then Oxygene and the RemObjects toolchain has evolved dramatically and is now capable of a lot more than CLR support.
- You can compile to raw, llvm optimized machine code for 8 platforms
- You can compile to CLR/.Net
- You can compile to Java bytecodes
- You can compile to WebAssembly!
WebAssembly is not Javascript, it’s important to underline that. WebAssembly was created especially for developers using traditional languages, so that traditional compilers can emit web friendly, binary code. Unlike Javascript, WebAssembly is a purely binary format. Just like Delphi generates machine-code that is linked into a final executable, WebAssembly is likewise compiled, linked and emitted in binary form.
If that sounds like a sales pitch, it’s not. It’s a matter of practicality.
- WebAssembly is completely barren out of the box. The runtime environment, be it V8 for the browser or V8 for node.js, gives you nothing out of the box. You don’t even have WriteLn() to emit text.
- Google expects compiler makers to provide their own RTL functions, from the fundamental to the advanced. The only thing V8 gives you, is a barebone way of referencing objects and functions on the other side, meaning the JS and DOM world. And that’s it.
So the reason i’m talking a lot about Oxygene and RemObjects Elements (Elements is the name of the compiler toolchain RemObjects offers), is because it ships with an RTL. So you are not forced to start on actual, literal assembly level.
If you don’t want to study JavaScript, Oxygene and Elements from RemObjects is the solution
RemObjects also delivers a DelphiVCL compatibility framework. This is a clone of the Delphi VCL / Freepascal LCL. Since WebAssembly is still brand new, work is being done on this framework on a daily basis, with updates being issued all the time.
Note: The Delphi VCL framework is not just for WebAssembly. It represents a unified framework that can work anywhere. So if you switch from WebAssembly to say Android, you get the same result.
The most important part of the above, is actually not the visual stuff. I mean, having HTML5 visual controls is cool – but chances are you want to use a library like Sencha, SwiftUI or jQueryUI to compose your forms right? Which means you just want to interface with the widgets in the DOM to set and get values.
You probably want to use a fancy UI library, like jQuery UI. This works perfectly with Elements because you can reference the controls from your WebAssembly module. You dont have to create TButton, TListbox etc manually
The more interesting stuff is actually the non-visual code you get access to. Hundreds of familiar classes from the VCL, painstakingly re-created, and usable from any of the 5 languages Elements supports.
You can check it out here: https://github.com/remobjects/DelphiRTL
Skipping JavaScript all together
I dont believe in single languages. Not any more. There was a time when all you needed was Delphi and a diploma and you were set to conquer the world. But those days are long gone, and a programmer needs to be flexible and have a well stocked toolbox.
Knowing where you want to be is half the journey
The world really don’t need yet-another-c# developer. There are millions of C# developers in India alone. C# is just “so what?”. Which is also why C# jobs pays less than Delphi or node.js system service jobs.
What you want, is to learn the things others avoid. If JavaScript looks alien and you feel uneasy about the whole thing – that means you are growing as a developer. All new things are learned by venturing outside your comfort zone.
How many in your company can write high quality WebAssembly modules?
How many within one hour driving distance from your office or home are experts at WebAssembly? How many are capable of writing industrial scale, production ready system services for node.js that can scale from a single instance to 1000 instances in a large, clustered cloud environment?
Any idiot can pick up node.js and knock out a service, but with your background from Delphi or C++ builder you have a massive advantage. All those places that can throw an exception that JS devs usually ignore? As a Delphi or Oxygene developer you know better. And when you re-apply that experience under a different language, suddenly you can do stuff others cant. Which makes your skills valuable.
The Quartex Media Desktop have made even experienced node / web developers gasp. They are not used to writing custom-controls and large-scale systems, which is my advantage
So would you learn JavaScript or just skip to WebAssembly? Honestly? Learn a bit of both. You don’t have to be an expert in JavaScript to compliment WebAssembly. Just get a cheap book, like “Node.js for beginners” and “JavaScript the good parts” ($20 a piece) and that should be more than enough to cover the JS side of things.
Adding WebAssembly to your resume and having the material to prove you know your stuff, is going to be a hell of a lot more valuable in the years to come than C#, Java or Python. THAT I can guarantee you.
And, we have a wicked cool group on Facebook you can join too: Click here to visit RemObjects Developer.
Enumerating network adapters in DWScript/Smart under Node.js
This is something I never had the time to implement under Smart Pascal, but it should be easy enough to patch. If you are using DWScript with the QTX Framework this is already in place. But for Smart users, here is a quick recipe.
First, we need access to the node.js OS module:
unit qtx.node.os;
//#############################################################################
// Quartex RTL for DWScript
// Written by Jon L. Aasenden, all rights reserved
// This code is released under modified LGPL (see license.txt)
//#############################################################################
unit NodeJS.os;
interface
uses
NodeJS.Core;
type
TCpusResultObjectTimes = class external
property user: Integer;
property nice: Integer;
property sys: Integer;
property idle: Integer;
property irq: Integer;
end;
TCpusResult = class external
property model: String;
property speed: Integer;
property times: TcpusResultObjectTimes;
end;
JNetworkInterfaceInfo = class external
property address: string;
property netmask: string;
property family: string;
property mac: string;
property scopeid: integer;
property internal: boolean;
property cidr: string;
end;
Jos_Exports = class external
public
function tmpDir: String;
function hostname: String;
function &type: String;
function platform: String;
function arch: String;
function release: String;
function uptime: Integer;
function loadavg: array of Integer;
function totalmem: Integer;
function freemem: Integer;
function cpus: array of TCpusResult;
function networkInterfaces: variant;
property EOL: String;
end;
function NodeJSOsAPI: Jos_Exports;
implementation
function NodeJSOsAPI: Jos_Exports;
begin
result := Jos_Exports(RequireModule("os") );
end;
end.
With that in place, we can start enumerating through the adapters. Remember that a PC can have several adapters attached, from a dedicated card to X number of USB wifi sticks.
Here is a little routine that goes through the adapters, and returns the first IPv4 LAN address it finds. This is very useful when writing servers, since you need the IP + port to setup a binding. And yes, you can just call HostName(), but the point here is to know how to run through the adapter array.
function GetMyV4LanIP: string;
begin
var OSAPI := NodeJSOsAPI();
var NetAdapters := OSAPI.networkInterfaces();
for var Adapter in NetAdapters do
begin
// Skip loopback device
if Adapter.Contains('Loopback') then
continue;
for var netIntf in NetAdapters[Adapter] do
begin
var address = JNetworkInterfaceInfo( NetAdapters[Adapter][netIntf] );
if not address.internal then
begin
// force copy of string
var lFam: string := string(address.family) + " ";
// make sure its ipv4
if lFam.ToLower().Trim() = 'ipv4' then
begin
result := address.address + " ";
result := result.trim();
break;
end;
end;
end;
end;
if result.length < 1 then
result := '127.0.0.1';
end;
Jon L. Aasenden 
You must be logged in to post a comment.