Archive
QuartexDeveloper.com is now active
It’s taken a while but Quartex Pascal now has it’s own website and forum. You can visit QuartexDeveveloper.com and check it out.
The SSL certificates are being applied within 72hrs according to the host, so don’t be alarmed that it shows up under HTTP rather than HTTPS right now – that is just temporary.
Up until now we have operated with a mix of donations and Patreon to help fund the project, but obviously that model doesn’t scale very well. After some debate and polls on the Facebook group I have landed on a new model.
Funding and access model
Starting with the release of version 1.0, which is just around the corner – the model will be as such:
- Backing and support will be handled solely through Patreon
- Facebook group will become open for all
- Patreon tiers will be modified to reflect new model
- Main activity and news will shift to our website, quartexdeveloper.com
- Community build will be available from our website
- Commercial license will also be available from our website
So to sum up, the following 3 options are available:
- Back the project on Patreon, full access to the latest and greatest 24/7
- Community edition, free for educational institutions and open-source projects (non commercial)
- Commercial license is for those that don’t want to back the project on a monthly basis, but instead use the community edition in a professional capacity for commercial work.
With the community edition available, why should anyone bother to back the project you might ask? Well, the public builds will by consequence be behind the latest, bleeding edge builds since the community edition is only updated on minor or major version increments (e.g. when version changes from 1.0 to 1.1). Users who back the project via Patreon will have instant access to new documentation, new packages with visual components, new project templates, RTL fixes and patches as they are released. These things will eventually trickle down to the community edition through version increments, but there is a natural delay involved.
This is how most modern crowd funded projects operate, with LTS builds (long term support) easily available while the latest cutting edge builds are backers only. Documentation, fixes and updates to components, new component packages, hotfixes and so on – is the incentive for backing the project.
This is the only way to keep the ball rolling without a major company backing day to day development, we have to get creative and work with what we got. Projects like Mono C# had the luxury of two major Linux distribution companies backing them, enabling Miguel de Icaza to work full time on the codebase. I must admit I was hoping Embarcadero would have stepped in by now, but either way we will get it done.
Onwards!
Quartex Pascal: Public Alpha
It’s been a long time since I have written a post here on my blog. I have been so busy with work that I quite frankly have not had the extra energy to maintain this website. During the weekdays my hands are full with work, and in the weekends I typically recharge 1 day, with Friday afternoon and Sunday allocated for the Quartex Pascal project. On Saturdays I sleep for as long as I can, go for a walk, and watch Netflix.
Thankfully I am happy to report that Quartex Pascal is more or less ready for a public alpha. I took out a week vacation today to finish the remaining handful of tickets, which are ultimately superficial and fiddly, but nothing difficult compared to what we have already achieved.
Have we reached our goals?
When you start a project it’s easy to get caught up in the potential. One feature quickly avails the next, and if you are not careful – you can be whisked away to vaporware land. Or even worse, end up with a project that never ends and where you keep telling yourself “i just need to add this, then I’m done”. I am happy that we have managed to avoid that, and set a clear boundary of what should be in the initial release. The point of version 1.0 is not to cover all possible features, but rather – to make damn sure the fundamental features work as well as I can make them. Because future revisions and features will build on that foundation. So in short: Yes. As much as 90% of what we set out to include in version 1.0 has been realized. The only thing we had to push to version 1.1 and 1.2 is the database explorer, DAC classes so your web application can work directly with a database through a node.js service, and a few minor features like having a Gr32 powered picture viewer and paint-program included.
There is plenty of room for optimization and refactoring in the code-base, so once the first version is out you can expect regular updates (Patreon backers only) where both the IDE and RTL becomes more and more refined and optimized. One of the things I am really looking forward to is writing new and exciting widgets (controls are called widgets under the QTX paradigm), and also port over more JS modules and frameworks. We already have an impressive list of JS frameworks that you can use out of the box. The benefit of having highly skilled backers is that they are quick to digest new technology and produce packages, so we have a lot of widgets that you can drag & drop that are 1:1 wrappers (a wrapper is a class definition that describes an external object, or a unit that makes the features of an external framework usable from pascal).
The Cloud desktop project
Since a couple of years have passed, most people have probably forgotten why Quartex Pascal was created to begin with. Namely as a development tool to implement and finish the Quartex Media Desktop (also known as Amibian.js). Quartex Pascal was actually a detour we had to make to save the codebase I had already implemented. So the moment version 1.0 goes out the door, my first priority is to refactor and re-implement the desktop client under the Quartex Pascal RTL. The background node.js services already run on my new RTL, so all the work we did a couple of years ago is still there, waiting to be picked up again.
I am not going to spend ages re-hashing the desktop system, but in short this is a client-server system that implements a Windows like desktop, complete with filesystem over websocket, multi user accounts, message based API, and that is 100% JavaScript from the back-end services all the way up to the desktop itself. It is in other words portable and completely hardware and platform agnostic. The point of the desktop is to provide the exact same ecosystem that Windows provides for native applications, for enterprise level web applications. This includes hybrid application modules where half the program is deployed server-side, while the visual part is rendered in the browser (this is how we could have a Torrent client with live status in a web application).
Combine this with a thin Linux bootstrap, where you boot into Chrome in Kiosk mode – and you have a fully working, incredibly powerful desktop system. One that you can literally copy from one machine to the next without recompiling a line of code. As long as the system supports node.js and have a modern browser, Amibian.js will run. Heck, I even booted it on my Smart TV (!).
Release date?
The public alpha is, as the name implies, a pre-release version meant purely to be played around with. There are bound to be hiccups and bugs, but the point is just to get you familiar with the ecosystem (which is very different from Delphi, so don’t think you can just magically compile some old Delphi application).
I am aiming at next weekend. It can be that some delay comes up, but all in all I have only a handful of tickets, most of them small and somewhat fiddly, but nothing too difficult. I will close 2 or 3 tickets a day, so a build should be ready next weekend for you guys.
What License?
The application is released as vanilla shareware, which means that copyright and ownership of all materials (except packages and examples written by others naturally) is tied to me. Once we have enough to establish the Quartex Pascal Foundation (which aims at teaching object pascal and offering free development tools for for students, schools and non-profit organizations) ownership will be isolated there. You can read more about the license on the website, here: https://quartexpascal.wordpress.com/about/licensing/.
We also have a rule that any version of Quartex Pascal will never cost more than €300. The first version will be in the €100-€150 range, which buys you a license to use the development tools in a commercial setting. Quartex Pascal is free to use for open-source work. Students can also use QTX for free, provided they provide proper student identification that can be verified, and they dont use it for commercial gain. Considering the cheap price, buying a license wont exactly break the bank.
The RTL is accessible the exact same way that you are used to under Delphi, so exploring the RTL is encouraged. I have started on the documentation but this is an alpha so you really need to explore a bit.
Just like Delphi all applications have a TApplication object that is the first to be created, and entities like forms are managed by TApplication (they automatically register when you create TQTXForm or TQTXWindow). Once you familiarize yourself with the units, you should have no problem becoming productive in a very short time.
Resources
- Check out the Patreon project page for backing this project
- Check out the Quartex Pascal website
- Check out the Quartex Pascal usergroup on Facebook (backers only!)
Quartex Pascal Build 13b ships
While it can come across as disingenuous, I frickin love this project! As a developer yourself you know that feeling, when you manage to unify various aspects of your program, so that it all fits just perfectly. And the way I implemented file-handling and editors is within that sweet spot.
What is new?
It’s been a couple of weeks since I posted here last, so the list of changes will be quite dramatic. I think its best to focus on the highlights or this post would become very long!
Ironwood license management
Up until 2018 one of my products was a component package called HexLicense. This is a component package for Delphi that provides serial number validation, license handling and (most importantly) serial number minting. The HexLicense components were sold commercially until 2018 when I took them off the market and open-sourced (access was via Patreon. It is now moved to the Quartex Pascal project instead).
Im not going to go into how difficult it is to produce thousands of distinctly different serial numbers based on seed data, but it’s no walk in the park.
The final implementation I made for license minting and validation, was called Ironwood. It took the engine behind HexLicense and took it to a completely new level, incorporating both obfuscation and number modulation.
Needless to say, Ironwood is now a part of the Quartex Pascal RTL. To make it easier to work with the IDE has a nice utility for generating license-numbers, loading and saving keys, exporting license number batches – and much more.
There is also a ready-to-rock node.js application that can generate keys from the command-line (which is good to invoke from a server or service, so that it executes as a separate process).
HTML structure provider
The IDE has a very clean internal architecture, where the actual work is isolated in a set of easy to understand classes. One of these classes is called a TIDEAstProvider class. This is a class whose job it is to parse and otherwise work with whatever content an editor has, and deliver symbolic information that can be displayed in the file-structure treeview.
Obviously we have an object pascal provider, which will quickly compile and generate an AST very quickly in memory. This is used to power both the structure treeview and the code-suggestion.
Next, we have the exact same provider for JavaScript. So when you open a JavaScript file, the file will be processed to produce an AST, and the symbol information will be displayed exactly like your object pascal is. So behavior between these are identical.
We now also have a HTML provider, with a CSS provider on the way. The HTML provider is still in its infancy, but its flexible enough to represent a good foundation to work with. So I will no doubt return to this task later to smarten the provider logic up, and better handle un-valid HTML and CSS.
Code suggestion
Code suggestion is a pretty standard function these days. We have had support for this under Object Pascal for a while now in the IDE (with JavaScript on the way).
Note: the code suggestion-box is un-styled at this point. Custom painting will be added once the core functionality is done.
Code suggestion for HTML is now in place too. It needs a bit of polish since the rules for HTML are wildly different from a programming language, but common behavior like TAG suggestion is there — with attributes, properties and events to follow.
So even if you are not an object pascal developer, the IDE should be nice to work with for traditional JavaScript / HTML code.
Form Recognition
While we cannot activate the form-designer just yet, since we need more AST functionality to extract things like class properties and attributes “live” to be able to do that properly — we are getting really close to that milestone.
The IDE however now recognize form files, so if your unit has an accompanying DFM file, the IDE is smart enough to open up a form-page. Form pages are different from ordinary pascal pages, since they also have the form designer control on a sub-tab. More or less identical to Delphi and Lazarus.
It is going to be so nice to get the form-designer activated. Especially the stack-based layout, which makes scalable, dynamic layout easy to create and work with.
The QTX RTL also supports orientation awareness as a part of the visual component system. One of the first things you will notice when exploring the code, is that ReSize() ships in an Orientation parameter, so you can adjust your layout accordingly.
Help and documentation inside the IDE
The IDE now has a PDF viewer with search functionality built-in. So when you click on Help and Documentation, a tab which shows the documentation PDF opens. This makes it easy to read, learn and find the information you need fast.
Well, that was a brief overview of what has changed since last time!
Next update is, as always, the weekends. We tend to land on sundays for new binaries, but do issue hotfixes in the evenings (weekdays) if something critical shows up.
Come join the fun!
Want to support the project? All financial backers that donates $100+ get their name in the product, access to the full IDE source-code on completion, and access to the Quartex Media Desktop system (which is a complete web desktop with a clustered back-end, compiled to JavaScript and running on node.js. Portable, platform and chipset independent, and very powerful).
A smaller sum monthly is also welcome. The project would not exist without members backing it with $100, $200 etc every month. This both motivates and helps me allocate hours for continuous work.
When the IDE is finished you will also have access to the IDE source-code through special dispensation. Backers have rights since they have helped create the project.
All donations are welcome, both large and small. But donations over $100, especially reoccurring, is what drives this project forward.
Remember to send me a message on Facebook so I can add you to the Admin group: https://www.facebook.com/quartexnor/
Quartex Pascal, status
With Quartex Pascal development at full steam and a dedicated Facebook group for the backers – It’s not often that I post updates here on my blog. One of the benefits of being a backer is that you have direct access to the latest builds, and also that you take part in the dialog on the group.
Be that as it may, here are some of the news happening with Quartex Pascal!
What’s new?
Quite a bit has happened since my last blog post. The IDE is coming together piece by piece, and at the moment i’m focusing on helper functionality for the AST (abstract symbol tree).
As you no doubt know, when a program is compiled it’s first parsed and converted into objects in memory. So every inch of your program becomes an elaborate tree-structure. This structure is what is commonly called the AST – and it is the raw material if you will, that code is generated from. In our case we don’t produce machine code, but rather JavaScript.
As you can imagine such an AST is quite complex. It has to be able to represent all the nuances of Object Pascal, as well as simplify finding information about everything. Every datatype, every record, class or complex structure, every field, expression — it all exists within the AST.
Without code to quickly traverse and work with the AST, things like code suggestions, parameter suggestions, code completion, the much loved ctrl + click — none of those would work. So while boring, it has to be done.
Oh and the mini-map has been re-implemented from scratch, so it’s now fast, accurate and responsive – and it works with mousewheel and keyboard.
Code suggestion
One of features that is bubbling up to the surface right now, is code suggestion. It’s something that we simply take for granted these days, and we dont really consider how much work it is to make. Eric has done a lot to simplify this for DWScript, but you still have to build up the codebase around it. But thankfully that is now largely done, leaving only a bit of styling and focus handling.
Form Files
In the previous build the IDE only recognized unit files (.pas), but in the current version it will check for an accompanying .DFM file. If a form-design file exists, it will open up a form-designer page rather than a pure code page.
The form-designer itself has received a bit of love lately too. Keyboard shortcuts have been added, such as holding down CTRL during a multiple selection — and changes to the layout is signaled back to the IDE and reflected in save-states changing (i.e if you change a form layout, the Save and Save All icon becomes enabled).
The form layout objects (visual widgets) have also been re-worked a bit. We want our DFM file-format to be JSON, so full JSON object persistence has been implemented. The form-designer widgets inherit from TQTXJSONPersistent, making it a literal one-liner to load and save form design.
We do need to wait for the AST explorer code to finish though, before you can start dragging & dropping widgets. Visual controls dont magically appear by themselves. Packages must be registered, and visual controls must likewise be registered with the IDE before they become known to the designer. So once the AST code is finished, we move on to packages – and finally glue the pieces together.
RTL advancements
The RTL has seen just as much changes as the IDE itself, and for good reason. Unlike “other” Web Technology tools, Quartex Pascal has an RTL that supports everything HTML5 has to offer. You are not limited to a static, fixed layout like we are used to under Delphi or Lazarus.
The ability to work with dynamic layout presents some interesting and highly efficient design opportunities. I find myself using the blocking layout model more and more, since it simplifies building up a dynamic UI that scales. Being able to work with different font scales too, like point (pt), as opposed to traditional pixel (px) closes the circle; it makes it possible to implement visual components that can scale it’s content to fit the container. This in turn simplifies writing software that renders well on both Desktop, Mobile and browser.
The changes has been too many to list here, but I have pretty much implemented all the event delegate objects (more to come), tweaked creation speed even further – and added additional polyfill files that ensures that your code works on every browser (a polyfill is a fallback system, so if a browser lacks a feature – the polyfill is used instead).
Application models
Under Quartex Pascal the TApplication object plays an important role, much more than you are used to under Delphi or Lazarus. It is TApplication that is responsible for maintaining your layout – and ultimately how forms are shown.
- If you are writing a mobile application you obviously want your forms to slide into view, just like native applications do on iPhone and Android.
- If you are writing a client-server website solution, you might prefer that forms covers the full width of the browser, with variable height – with the user switching forms by clicking on a toolbar, menu option or link.
- Perhaps you would like the forms to the stacked vertically, so that each form comes into view as the user scrolls downwards – perhaps with some fancy effect, or a static background behind the forms.
- And last but not least, you might prefer that your web application looks and behaves like a Windows desktop application. With multiple windows that can be moved around, a normal menu system on top of each window – or on top of the browser.
The only way to consolidate these diverse and even conflicting layout models, is to implement several TApplication classes; each one representing the layout model you want to work with. So when your create a project, you pick the layout model you want – and the correct TApplication is chosen and generated for your project.
Actual menus
The RTL have seen a few new widgets added, but the most interesting one is without a doubt the Menu widget. This is a widget that mimics how a normal menu works in a real program.
Creating a menu might not sound interesting, but it’s actually a small challenge under HTML. Not the coding itself, but dealing with menu presentation without visual artifacts. Whenever you click a menu that has a sub-menu attached, the new menu is created from code dynamically. It’s positioned at the end of it’s invoker (to the right of the parent menu item) and should only show up when all it’s child elements have been created.
This was very tricky to get right under a competing system, because the way elements was created was, well, wrong. You want to avoid reflows at all cost during the constructor – otherwise there will be visual artifacts and flickering. But that is not an issue under QTX. And the speed is insane. Even with 100 recursive items on a menu container, it’s virtually instantaneous.
If you are wondering why this makes any difference, you have to remember Quartex Media Desktop. This is not a simple toy with an onClick event, but can be bound into the process tree of the media desktop. The new code is barely 500 lines of code, the older version was over 3000 lines of code.
The goal for the IDE is that you can create a full desktop as a project. Not just programs that should run on the desktop (and its Ragnarok message protocol interface) – but the actual desktop system, which also covers several node.js system services.
The reason this is cool is because this enables you to deliver full scale, desktop level software purely through the browser. Such a desktop would be suitable for a school, a tutoring company, as an intranet – or for teams that need to share files, chat in realtime — and do their software development via the same web interface.
So it’s “a little bit” bigger than some mock desktop.
Come join the fun!
Want to support the project? All financial backers that donates $100+ get their name in the product, access to the full IDE source-code on completion, and access to the Quartex Media Desktop system (which is a complete web desktop with a clustered back-end, compiled to JavaScript and running on node.js. Portable, platform and chipset independent, and very powerful).
A smaller sum monthly is also welcome. The project would not exist without members backing it with $100, $200 etc every month. This both motivates and helps me allocate hours for continuous work.
When the IDE is finished you will also have access to the IDE source-code through special dispensation. Backers have rights since they have helped create the project.
All donations are welcome, both large and small. But donations over $100, especially reoccurring, is what drives this project forward.
Remember to send me a message on Facebook so I can add you to the Admin group: https://www.facebook.com/quartexnor/
Quartex Pascal, Build 10b is out for backers
I am deeply moved by some of the messages I have received about Quartex Pascal. Typically from people who either bought Smart Mobile studio or have followed my blog over the years. In short, developers that like to explore new ideas; people who also recognize some of the challenges large and complex run-time libraries like the VCL, FMX and LCL face in 2020.
Since I started with all this “compile to JavaScript” stuff, the world has changed. And while I’m not always right – I was right about this. JavaScript will and have evolved into a power-house. Largely thanks to Microsoft killing Basic. But writing large scale applications in JavaScript, that is extremely time consuming — which is where Quartex Pascal comes in.
Quartex Pascal evolves every weekend. There are at least 2 builds each weekend for backers. Why not become a backer and see the product come to life? Get instant access to new builds, the docs, and learn why QTX code runs so much faster than the alternatives?
A very important distinction
Let me first start by iterating what I mentioned in my previous post, namely that I am no longer involved with The Smart Company. Nor am I involved with Smart Mobile Studio. I realize that it can be difficult for some to separate me from that product, since I blogged and created momentum for it for more than a decade. But alas, life change and sometimes you just have to let go.
The QTX Framework which today has become a fully operational RTL was written by me back between 2013-2014 (when I was not working for the company due to my spinal injury). And the first version of that framework was released under an open-source license.
When I returned to The Smart Company, it was decided that to save time – we would pull the QTX Framework into the Smart RTL. Since I owned the QTX Framework, and it was open source, it was perfectly fine to include the code. The code was bound by the open source licensing model, so we broke no rules including it. And I gave dispensation that it could be included (although the original license explicitly stated that the units should remain untouched and separate, and only inherited from).
Quartex Media Desktop, a complete desktop system (akin to X Windows for Linux) written completely in Object Pascal, including a clustered, service oriented back-end. All of it done in Quartex Pascal — a huge project in its own right. Under Quartex Pascal, this is now a project type, which means you can have your own cloud solution at the click of a button.
As I left the company for good before joining Embarcadero, The Smart Company and myself came to an agreement that the parts of QTX that still exists in the Smart Mobile Studio RTL, could remain. It would be petty and small to make a huge number out of it, and I left on my own terms. No point ruining all that hard work we did. So we signed an agreement that underlined that there would be overlaps here and there in our respective codebases, and that the QTX Framework and Quartex Media Desktop was my property.
Minor overlaps
As mentioned there will be a few minor overlaps, but nothing substantial. The class hierarchy and architecture of the QTX RTL is so different, that 90% of the code in the Smart RTL simply won’t work. And I made it that way on purpose so there would be no debates over who made what.
QTX represents how I felt the RTL should have been done. And its very different from where Smart Mobile Studio ended up.
The overlaps are simple and few, but it can be helpful for Smart developers to know about if they plan on taking QTX for a test-drive:
- TInteger, TString and TVariant. These were actually ported from Delphi (part of the Sith Library, a pun on Delphi’s Jedi Library).
- TDataTypeConverter came in through the QTX Framework. It has been completely re-written from scratch. The QTX version is endian aware (works on both ARM, X86 and PPC). Classes that deal with binary data (like TStream, TBuffer etc) inherit from TDataTypeConverter. That way, you dont have to call a secondary instance just to perform conversion. This is easier and much more efficient.
- Low-level codecs likewise came from the QTX Framework, but I had to completely re-write the architecture. The old model could only handle binary data, while the new codec classes also covers text based formats. Codecs can be daisy-chained so you can do encoding, compression and encryption by feeding data into the first, and catching the processed data from the last codec in the chain. Very handy, especially when dealing with binary messages and database drivers.
- The in-memory dataset likewise came from the QTX Framework. This is probably the only unit that has remained largely unchanged. So that is a clear overlap between the Smart RTL and QTX.
- TextCraft is an open source library that covers Delphi, Freepascal and DWScript. The latter was pulled in and used as the primary text-parser in Smart. This is also the default parser for QTX, and have been largely re-written so it could be re-published under the Shareware license.
Since the QTX RTL is very different from Smart, I haven’t really bothered to use all of the old code. Stuff like the CSS Effects units likewise came from the QTX Framework, but the architecture I made for Smart is not compatible with QTX, so it makes no sense using that code. I ported my Delphi tweening library to DWScript in 2019, which was a part of my Patreon project. So most of the effects in QTX use our own tweening library. This has some very powerful perks, like being able to animate a property on any object rather than just a HTML Element. And you can use it for Canvas drawing too, which is nice.
Progress. Where are we now?
So, where am I in this work right now? The RTL took more or less 1 year to write from scratch. I only have the weekends for this type of work, and it would have been impossible without my backers. So I cannot thank each backer enough for the faith in this. The RTL and new IDE is actually just a stopping point on the road to a much bigger project, namely CloudForge, which is the full IDE running as an application on the Quartex Media Desktop. But first, let’s see what has been implemented!
AST unit view
The Unit Overview panel. Easy access to ancestor classes as links (still early R&D). And the entire RTL on a second tab. This makes it very easy to learn the new RTL. There is also proper documentation, both as PDF and standard helpfile.
When the object-pascal code is compiled by DWScript, it goes through a vigorous process of syntax checking, parsing, tokenizing and symbolization (or objectification is perhaps a better word), where every inch of the code is transformed into objects that the compiler can work with and produce code from. These symbols are isolated in what is known as an AST, short for “Abstract Symbol Tree”. Basically a massive in-memory tree structure that contains your entire program reduced to symbols and expressions.
In order for us to have a live structural view of the current unit, I have created a simple background process that compiles the current unit, grabs the resulting AST, locates the unit symbol, and then displays the information in a meaningful way. This is the exact same as most other IDE’s do, be it Visual Studio, Embarcadero Delphi, or Lazarus.
So we have that in place already. I also want to make it more elaborate, where you can click yourself to glory by examining ancestors, interfaces, partial class groups – as well as an option to include inherited members (which should be visually different). Either way, the AST code is done. Just need to consolidate a few tidbits so that each Treeview node retains information about source-code location (so that when you double-click a symbol, the IDE navigates to where the symbol exists in the codebase).
JavaScript parsing and compilation
QTX doesn’t include just one compiler, but three. In order for the unit structure to also work for JavaScript files I have modified Besen, which is an ES5 compatible JavaScript engine written in Delphi. And I use this much like DWScript to parse and work with the AST.
Besen is a wonderful compiler. Where DWScript produces JavaScript from Object Pascal, Besen produces bytecodes from JavaScript (which are further JIT compiled). This opens up for some interesting options. I need to add support for ES6 though, modules and require are especially important for modern node.js programming (and yes, the QTX RTL supports these concepts)
HTML5 Rendering and CSS preview
Instead of using Chromium inside the IDE, which is pretty demanding, I have decided to go for HTMLComponents to deal with “normal” tasks. The “Welcome” tab-page for example — it would be complete overkill to use a full Chromium instance just for that, and TEdgeBrowser is likewise shooting sparrows with a Bazooka.
THTMLComponents have a blistering fast panel control that can render more or less any HTML5 document you throw at it (much better than the old TFrameViewer component). But obviously, it doesn’t have JS support. But we won’t be using JS when displaying static information – or indeed, editing HTML5 compliant content.
WYSIWYG Editor
The biggest benefit for HTMLComponents, is that it’s a fully operational HTML compliant editor. Which means you can do more or less all your manual design with that editor. In Quartex Pascal there is direct support for HTML files. Quartex works much like Visual Studio code, except it has visual designers. So you can create a HTML file and either type in the code manually, or switch to the HTMLComponents editor.
Which is what products like Help & Manual uses it for
Image from HTMLComponents application gallery website
Support for HTML, CSS and JS files directly
While not new, this is pretty awesome. Especially since we can do a bit of AST navigation here too to present similar information as we do for Object Pascal. The whole concept behind the QTX RTL, is that you have full control over everything. You can stick to a normal Delphi like form designer and absolute positioning, or you can opt for a more dynamic approach where you create content via code. This is perfect for modern websites that blend scrolling, effects and content (both dynamic and static content) for a better user experience.
You can even (spoiler alert), take a piece of HTML and convert it into visual controls at runtime. That is a very powerful function, because when doing large-scale, elaborate custom controls – you can just tell the RTL “hey, turn this piece of HTML into a visual control for me, and deliver it back when you are ready).
Proper Form Designer
Writing a proper form designer like Delphi has is no walk in the park. It has to deal not just with a selected control, but also child elements. It also has to be able to select multiple elements based on key-presses (shift + click adds another item to the selection), or from the selection rectangle.
A property form layout control. Real-time rendering of controls is also possible, courtesy of HTMLComponents. But to be honest, it just gets in the way. Its much easier to work with this type of designer. It’s fast, responsive, accurate and will have animated features that makes it a joy to work with.
Well, that’s not going to be a problem. I spent a considerable amount of time writing a proper form designer, one that takes both fixed and dynamic content into account. So the Quartex form designer handles both absolute and stacked layout modes (stacked means top-down, what in HTML is knock as blocking element display, where each section stretch to the full width, and only have a defined height [that you can change]).
Node.js Service Protocol Designer
Writing large-scale servers, especially clustered ones, is very fiddly in vanilla JavaScript under node.js. It takes 3 seconds to create a server object, but as we all know, without proper error handling, a concurrent message format, modern security and a solid framework to handle it all — that “3 second” thing falls to the ground quickly.
This is where the Ragnarok message system comes in. Which is both a message framework, and a set of custom servers adapted for dealing with that type of data. It presently supports WebSocket, TCP and UDP. But expanding that to include REST is very easy.
This is where the full might of the QTX Framework begins to dawn. As i wrote before we started on the Quartex Media Desktop (Which this IDE and RTL is a part of), in the future developers wont just drag & drop components on a form; they will drag & drop entire ecosystems ..
But the power of the system is not just in how it works, and how you can create your own protocols, and then have separate back-end services deal with one part of your infrastructure’s workload. It is because you can visually design the protocols using the Node Builder. This is being moved into the QTX IDE as I type. So should make it for Build 12 next weekend.
In short, you design your protocols, messages and types – a bit like RemObjects SDK if you have used that. And the IDE generates both server and client code for you. All you have to do is fill in the content that acts on the messages. Everything else is handled by the server components.
Suddenly, you can spend a week writing a large-scale, platform agnostic service stack — that would have taken JavaScript developers months to complete. And instead of having to manage a 200.000 lines codebase in JavaScript — you can enjoy a 4000 line, easily maintainable Object Pascal codebase.
Build 11
Im hoping to have build 11 out tomorrow (Sunday) for my backers. Im still experimenting a bit with the symbol information panel, since I want it to be informative not just for classes, but also for methods and properties. Making it easy to access ancestor implementations etc.
I also need to work a bit more on the JS parsing. Under ES5 its typical to use variables to hold objects (which is close to how we think of a class), so composite and complex datatypes must be expanded. I also need to get symbol position to work property, because since Besen is a proper bytecode compiler, it doesn’t keep as much information in it’s AST as DWScript does.
Widgets (which is what visual controls are called under QTX) should appear in build 12 or 13. The IDE supports zip-packages. The file-source system I made for the TVector library (published via Embarcadero’s website a few months back) allows us to mount not just folders as a file-source, but also zip files. So QTX component packages will be ordinary zip-files containing the .pas files, asset files and a metadata descriptor file that tells the IDE what to expect. Simple, easy and very effective.
Support the project!
Want to support the project? All financial backers that donates $100+ get their name in the product, access to the full IDE source-code on completion, and access to the Quartex Media Desktop system (which is a complete web desktop with a clustered back-end, compiled to JavaScript and running on node.js. Portable, platform and chipset independent, and very powerful).
Your help matters! It pays for components, hours and above all, tools and motivation. In return, you get full access to everything and a perpetual license. No backers will ever pay a cent for any future version of Quartex Pascal. Note: PM me after donating so I can get you added to the admin group! Click here to visit paypal: https://www.paypal.com/paypalme/quartexNOR
All donations are welcome, both large and small. But donations over $100, especially reoccurring, is what drives this project forward. It also gets you access to the Quartex Developer group on Facebook where new builds, features etc is happening. It is the best way to immediately get the latest build, read documentation as its written and see the product come to life!
Quartex Pascal, convergence is near
A Quartex Cluster of 5 x ODroid XU4. A $400 super computer running Quartex media Desktop. Enough to power a school.
I only have the weekends to work on Quartex Pascal, but I have spent the past 18 months tinkering away, making up for wasted time. So I’m just going to leave some pictures here for you to enjoy.
Note: I was asked on LinkedIn if this has anything to do with Smart Mobile Studio, and the answer is a resounding no. I have nothing to do with Smart any more. QTX Pascal is a completely separate project that is written from scratch by yours truly.
The QTX Framework was initially a library I created back in 2014, but it has later been completely overhauled and turned into a full RTL. It is not compatible with Smart Pascal and has a completely different architecture.
QTX Pascal is indirectly funded by the Amiga Retro Community (which might sound strange, but the technical level of that community is beyond anything I have encountered elsewhere) since QTX is central to the creation of the Quartex Media Desktop. It is a shame that Embarcadero decided to not back the project. The compiler and toolchain would have been a part of Delphi by now, and I wouldn’t have to write a separate IDE. But when they see what this system can deliver in terms of services, database work, mobile and embedded -they might regret it. The project only accepts donation funding, I am not interested in investors or partners. If you want a vision turned into reality, you gotta do it yourself. Everything else just gets in the way.
For developers by developers
Quartex Pascal is made for the community. It will be free for students and open-source projects. And a commercial license will never exceed $300. It is a shareware license and the financial aspects is purely to help fund further research and development of the desktop cloud platform. The final goal (CloudForge) is to compile the IDE itself to JavaScript, so people only need a browser to write enterprise level applications via Quartex Media Desktop. When that is finished, my work is done – and people have a clear path to the future.
Unlike other systems, QTX started with the non-visual stuff, so the system has a well implemented infrastructure for writing universal services and servers, using node.js as a deployment host. Services are also Docker friendly. Runs without change on Windows, Mac OS, Linux and a wealth of embedded systems and SBCs (single board computers)
A completely new RTL written from scratch generates close to native speed JS, highly compatible (even with legacy browsers) and rock solid
There are several display modes for QTX forms, from dynamic to absolute positioning. You can mix and match between HTML and QTX code, including a HTML5 compliant WYSIWYG editor and style manager. Makes content handling a lot easier
Write object pascal, JavaScript, HTML, LDEF (webassembly), node.js services – or mix and match between them all for maximum potential. Writing mobile applications is now ridiculously easy compared to “other tools” out there.
Oh and for the proverbial frosting — The full clustered Quartex Media desktop and services is a project type. Thats right. A complete cloud infrastructure suitable for teams, kiosks, embedded, schools, intranets – and even an replacement OS for ChromeOS. You don’t need to interface with Amazon, you get your own Amazon (optional naturally).
Filesystem over websocket, IPC between hosted apps and desktop, full back-end services that are clustered, and run on anything from a Raspberry PI 4 to low-cost ARM SBCs.
Web Assembly made easy. Both for Delphi and QTX
Let there be rock
Oh, and documentation. Loads and loads of documentation.
Proper documentation, both class overview and explanations that a human being has written is paramount for learning and getting up to speed quickly.
I don’t have vacation this year, which means I only have weekends to tinker away. But i have spent the past 18-ish months preparing and slowly finishing the pieces I needed. From vector containers to form design controls, to a completely re-written RTL from scratch — so yeah. This time I’m doing it my way.
Nodebuilder, QTX and the release of my brand new social platform
First, let me wish everyone a wonderful new year! With xmas and the silly season firmly behind us, and my batteries recharged – I feel my coding fingers itch to get started again.
2019 was a very busy year for me. Exhausting even. I have juggled both a full time job, kids and family, as well as our community project, Quartex Media Desktop. And since that project has grown considerably, I had to stop and work on the tooling. Which is what NodeBuilder is all about.
I have also released my own social media platform (see further down). This was initially scheduled for Q4 2020, but Facebook pissed me off something insanely, so I set it up in december instead.
NodeBuilder
For those of you that read my blog you probably remember the message system I made for the Quartex Desktop, called Ragnarok? This is a system for dealing with message dispatching and RPC, except that the handling is decoupled from the transport medium. In other words, it doesnt care how you deliver a message (WebSocket, UDP, REST), but rather takes care of serialization, binary data and security.
All the back-end services that make up the desktop system, are constructed around Ragnarok. So each service exposes a set of methods that the core can call, much like a normal RPC / SOAP service would. Each method is represented by a request and response object, which Ragnarok serialize to a JSON message envelope.
In our current model I use WebSocket, which is a full duplex, long-term connection (to avoid overhead of having to connect and perform a handshake for each call). But there is nothing in the way of implementing a REST transport layer (UDP is already supported, it’s used by the Zero-Config system. The services automatically find each other and register, as long as they are connected to the same router or switch). For the public service I think REST makes more sense, since it will better utilize the software clustering that node.js offers.
Node Builder is a relatively simple service designer, but highly effective for our needs
Now for small services that expose just a handful of methods (like our chat service), writing the message classes manually is not really a problem. But the moment you start having 20 or 30 methods – and need to implement up to 60 message classes manually – this approach quickly becomes unmanageable and prone to errors. So I simply had to stop before xmas and work on our service designer. That way we can generate the boilerplate code in seconds rather than days and weeks.
While I dont have time to evolve this software beyond that of a simple service designer (well, I kinda did already), I have no problem seeing this system as a beginning of a wonderful, universal service authoring system. One that includes coding, libraries and the full scope of the QTX runtime-library.
In fact, most of the needed parts are in the codebase already, but not everything has been activated. I don’t have time to build both a native development system AND the development system for the desktop.
NodeBuilder already have a fully functional form designer and code editor, but it is dormant for now due to time restrictions. Quartex Media Desktop comes first
But right now, we have bigger fish to fry.
Quartex Media Desktop
We have made tremendous progress on our universal desktop environment, to the point where the baseline services are very close to completion. A month should be enough to finish this unless something unforeseen comes up.
Quartex Media Desktop provides an ecosystem for advanced web applications
You have to factor in that, this project has only had weekends and the odd after work hours allocated for it. So even though we have been developing this for 12 months, the actual amount of days is roughly half of that.
So all things considered I think we have done a massive amount of code in such a short time. Even simple 2d games usually take 2 years of daily development, and that includes a team of at least 5 people! Im a single developer working in my spare time.
So what exactly is left?
The last thing we did before xmas was upon us, was to throw out the last remnants of Smart Mobile Studio code. The back-end services are now completely implemented in our own QTX runtime-library, which has been written from scratch. There is not a line of code from Smart Mobile Studio in QTX, which means we no longer have to care what that system does or where it goes.
To sum up:
- Push all file handling code out of the core
- Implement file-handling as it’s own service
Those two steps might seem simple enough, but you have to remember that the older code was based on the Linux path system, and was read-only.
So when pushing that code out of the core, we also have to add all the functionality that was never implemented in our prototype.
Each class actually represents a separate “mini” program, and there are still many more methods to go before we can put this service into production.
Since Javascript does not support threads, each method needs to be implemented as a separate program. So when a method is called, the file/task manager literally spawns a new process just for that task. And the result is swiftly returned back to the caller in async manner.
So what is ultimately simple, becomes more elaborate if you want to do it right. This is the price we pay for universality and a cluster enabled service-stack.
This is also why I have put the service development on pause until we have finished the NodeBuilder tooling. And I did this because I know by experience that the moment the baseline is ready, both myself and users of the system is going to go “oh we need this, and that and those”. Being able to quickly design and auto-generate all the boilerplate code will save us months of work. So I would rather spend a couple of weeks on NodeBuilder than wasting months having to manually write all that boilerplate code down the line.
What about the QTX runtime-library?
Writing an RTL from scratch was not something I could have anticipated before we started this project. But thankfully the worst part of this job is already finished.
The RTL is divided into two parts:
Non Visual code. Classes and methods that makes QTX an RTL- Visual code. Custom Controls + standard controls (buttons, lists etc)
Visual designer
As you can see, the non-visual aspect of the system is finished and working beautifully. It’s a lot faster than the code I wrote for Smart Mobile Studio (roughly twice as fast on average). I also implemented a full visual designer, both as a Delphi visual component and QTX visual component.
Quartex Media Desktop makes running on several machines [cluster] easy and seamless
Why is the QTX Runtime-Library important again?
When the desktop is out the door, the true work begins! The desktop has several roles to play, but the most important aspect of the desktop – is to provide an ecosystem capable of hosting web based applications. Offering features and methods traditionally only found in Windows, Linux or OS X. It truly is a complete cloud system that can scale from a single affordable SBC (single board computer), to a high-end cluster of powerful servers.
Clustering and writing distributed applications has always been difficult, but Quartex Media Desktop makes it simple. It is no more difficult for a user to work on a clustered system, as it is to work on a normal, single OS. The difficult part has already been taken care of, and as long as people follow the rules, there will be no issues beyond ordinary maintenance.
And the first commercial application to come out of Quartex Components, is Cloud Forge, which is the development system for the platform. It has the same role as Visual Studio for Windows, or X Code for Apple OS X.
The Quartex Media Desktop Cluster cube. A $400 super computer
I have prepared 3 compilers for the system already. First there is C/C++ courtesy of Clang. So C developers will be able to jump in and get productive immediately. The second compiler is freepascal, or more precise pas2js, which allows you to compile ordinary freepascal code (which is highly Delphi compatible) to both JavaScript and WebAssembly.
And last but not least, there is my fork of DWScript, which is the same compiler that Smart Mobile Studio uses. Except that my fork is based on the absolute latest version, and i have modified it heavily to better match special features in QTX. So right out of the door CloudForge will have C/C++, two Object Pascal compilers, and vanilla Javascript and typescript. TypeScript also has its own WebAssembly compiler, so doing hard-core development directly in a browser or HTML5 viewport is where we are headed.
Once the IDE is finished I can finally, finally continue on the LDEF bytecode runtime, which will be used in my BlitzBasic port and ultimately replace both clang, freepascal and DWScript. As a bonus it will emit native code for a variety of systems, including x86, ARM, 68k [including 68080] and PPC.
This might sound incredibly ambitious, if not impossible. But what I’m ultimately doing here -is moving existing code that I already have into a new paradigm.
The beauty of object pascal is the sheer size and volume of available components and code. Some refactoring must be done due to the async nature of JS, but when needed we fall back on WebAssembly via Freepascal (WASM executes linear, just like ordinary native code does).
A brand new social platform
During december Facebook royally pissed me off. I cannot underline enough how much i loath A.I censorship, and the mistakes that A.I does – in which you are utterly powerless to complain or be heard by a human being. In my case i posted a gif from their own mobile application, of a female body builder that did push-ups while doing hand-stands. In other words, a completely harmless gif with strength as the punchline. The A.I was not able to distinguish between a leotard and bare-skin, and just like that i was muted for over a week. No human being would make such a ruling. As an admin of a fairly large set of groups, there are many cases where bans are the result. Disgruntled members that acts out of revenge and report technical posts about coding as porn or offensive. Again, you are helpless because there are nobody you can talk to about resolving the issue. And this time I had enough.
It was always planned that we would launch our own social media platform, an alternative to Facebook aimed at adult geeks rather than kids (Facebook operates with an age limit of 12 years). So instead of waiting I rushed out and set up a brand new social network. One where those banale restrictions Facebook has conditioned us with, does not apply.
Just to underline, this is not some simple and small web forum. This is more or less a carbon copy of Facebook the way it used to be 8-9 years ago. So instead of having a single group on facebook, we can now have as many groups as we like, on a platform that looks more or less identical to Facebook – but under our control and human rules.
Amigadisrupt.com is a brand new social media platform for geeks
You can visit the site right now at https://www.amigadisrupt.com. Obviously the major content on the platform right now is dominated by retro computing – but groups like Delphi Developer and FPC developer has already been setup and are in use. But if you are expecting thousands of active users, that will take time. We are now closing in on 250 active users which is pretty good for such a short period of time. I dont want a platform anywhere near as big as FB. The goal is to get 10k users and have a stable community of coders, retro geeks, builders and creative individuals.
AD (Amiga Disrupt) will be a standard application that comes with Quartex Media Desktop. This is the beauty of web technology, in that it can unify different resources under one roof. And we will have our cake and eat it come hell or high water.
Disclaimer: Amiga Disrupt has a lower age limit of 18 years. This is a platform meant for adults. Which means there will be profanity, jokes that would get you banned on Facebook and content that is not meant for kids. This is hacker-land, and political correctness is considered toilet paper. So if you need social toffery like FB and Twitter deals with, you will be kicked by one of the admins.
After you sign up your feed will be completely empty. Here is how to get it started. And feel free to add me to your friends-list!
Quartex “Cloud Ripper” hardware
For close to a year now I have been busy on a very exciting project, namely my own cloud system. While I have written about this project quite a bit these past months, mostly focusing on the software aspect, not much has been said about that hardware.
Quartex “Cloud Ripper” running neatly on my home-office desk
So let’s have a look at Cloud Ripper, the official hardware setup for Quartex Media Desktop.
Tiny footprint, maximum power
Despite its complexity, the Quartex Media Desktop architecture is surprisingly lightweight. The services that makes up the baseline system (read: essential services) barely consume 40 megabytes of ram per instance (!). And while there is a lot of activity going on between these services -most of that activity is message-dispatching. Sending messages costs practically nothing in cpu and network terms. This will naturally change the moment you run your cloud as a public service, or setup the system in an office environment for a team. The more users, the more signals are shipped between the processes – but with the exception of reading and writing large files, messages are delivered practically instantaneous and hardly use CPU time.
Quartex Media Desktop is based on a clustered micro-service architecture
One of the reasons I compile my code to JavaScript (Quartex Media Desktop is written from the ground up in Object Pascal, which is compiled to JavaScript) has to do with the speed and universality of node.js services. As you might know, Node.js is powered by the Google V8 runtime engine, which means the code is first converted to bytecodes, and further compiled into highly optimized machine-code [courtesy of llvm]. When coded right, such Javascript based services execute just as fast as those implemented in a native language. There simply are no perks to be gained from using a native language for this type of work. There are however plenty of perks from using Node.js as a service-host:
- Node.js delivers the exact same behavior no matter what hardware or operating-system you are booting up from. In our case we use a minimal Linux setup with just enough infrastructure to run our services. But you can use any OS that supports Node.js. I actually have it installed on my Android based Smart-TV (!)
- We can literally copy our services between different machines and operating systems without recompiling a line of code. So we don’t need to maintain several versions of the same software for different systems.
- We can generate scripts “on the fly”, physically ship the code over the network, and execute it on any of the machines in our cluster. While possible to do with native code, it’s not very practical and would raise some major security concerns.
- Node.js supports WebAssembly, you can use the Elements Compiler from RemObjects to write service modules that executes blazingly fast yet remain platform and chipset independent.
The Cloud-Ripper cube
The principal design goal when I started the project, was that it should be a distributed system. This means that instead of having one large-service that does everything (read: a typical “native” monolithic design), we instead operate with a microservice cluster design. Services that run on separate SBC’s (single board computers). The idea here is to spread the payload over multiple mico-computers that combined becomes more than the sum of their parts.
Cloud Ripper – Based on the Pico 5H case and fitted with 5 x ODroid XU4 SBC’s
So instead of buying a single, dedicated x86 PC to host Quartex Media Desktop, you can instead buy cheap, off-the-shelves, easily available single-board computers and daisy chain them together. So instead of spending $800 (just to pin a number) on x86 hardware, you can pick up $400 worth of cheap ARM boards and get better network throughput and identical processing power (*). In fact, since Node.js is universal you can mix and match between x86, ARM, Mips and PPC as you see fit. Got an older PPC Mac-Mini collecting dust? Install Linux on it and get a few extra years out of these old gems.
(*) A single XU4 is hopelessly underpowered compared to an Intel i5 or i7 based PC. But in a cluster design there are more factors than just raw computational power. Each board has 8 CPU cores, bringing the total number of cores to 40. You also get 5 ARM Mali-T628 MP6 GPUs running at 533MHz. Only one of these will be used to render the HTML5 display, leaving 4 GPUs available for video processing, machine learning or compute tasks. Obviously these GPUs won’t hold a candle to even a mid-range graphics card, but the fact that we can use these chips for audio, video and computation tasks makes the system incredibly versatile.
Another design goal was to implement a UDP based Zero-Configuration mechanism. This means that the services will find and register with the core (read: master service) automatically, providing the machines are all connected to the same router or switch.
Put together your own supercomputer for less than $500
The first “official” hardware setup is a cluster based on 5 cheap ARM boards; namely the ODroid XU4. The entire setup fits inside a Pico Cube, which is a special case designed to house this particular model of single board computers. Pico offers several different designs, ranging from 3 boards to a 20 board super-cluster. You are not limited ODroid XU4 boards if you prefer something else. I picked the XU4 boards because they represent the lowest possible specs you can run the Quartex Media Desktop on. While the services themselves require very little, the master board (the board that runs the QTXCore.js service) is also in charge of rendering the HTML5 display. And having tested a plethora of boards, the ODroid XU4 was the only model that could render the desktop properly (at that low a price range).
Note: If you are thinking about using a Raspberry PI 3B (or older) as the master SBC, you can pretty much forget it. The media desktop is a piece of very complex HTML5, and anything below an ODroid XU4 will only give you a terrible experience (!). You can use smaller boards as slaves, meaning that they can host one of the services, but the master should preferably be an ODroid XU4 or better. The ODroid N2 [with 4Gb Ram] is a much better candidate than a Raspberry PI v4. A Jetson Nano is an even better option due to its extremely powerful GPU.
Booting into the desktop
One of the things that confuse people when they read about the desktop project, is how it’s possible to boot into the desktop itself and use Quartex Media Desktop as a ChromeOS alternative?
How can a “cloud platform” be used as a desktop alternative? Don’t you need access to the internet at all times? If it’s a server based system, how then can we boot into it? Don’t we need a second PC with a browser to show the desktop?
Accessing the desktop like a “web-page” from a normal Linux setup
To make a long story short: the “master” in our cluster architecture (read: the single-board computer defined as the boss) is setup to boot into a Chrome browser display under “kiosk mode”. When you start Chrome in kiosk mode, this removes all traces of the ordinary browser experience. There will be no toolbars, no URL field, no keyboard shortcuts, no right-click popup menus etc. It simply starts in full-screen and whatever HTML5 you load, has complete control over the display.
What I have done, is to to setup a minimal Linux boot sequence. It contains just enough Linux to run Chrome. So it has all the drivers etc. for the device, but instead of starting the ordinary Linux Desktop (X or Wayland) -we instead start Chrome in kiosk mode.
Booting into the same desktop through Chrome in Kiosk Mode. In this mode, no Linux desktop is required. The Linux boot sequence is altered to jump straight into Chrome
Chrome is started to load from 127.0.0.1 (this is a special address that always means “this machine”), which is where our QTXCore.js service resides that has it’s own HTTP/S and Websocket servers. The client (HTML5 part) is loaded in under a second from the core — and the experience is more or less identical to starting your ChromeBook or NAS box. Most modern NAS (network active storage) devices are much more than a file-server today. NAS boxes like those from Asustor Inc have HDMI out, ships with a remote control, and are designed to act as a media center. So you connect the NAS directly to your TV, and can watch movies and listen to music without any manual conversion etc.
In short, you can setup Quartex Media Desktop to do the exact same thing as ChromeOS does, booting straight into the web based desktop environment. The same desktop environment that is available over the network. So you are not limited to visiting your Cloud-Ripper machine via a browser from another computer; nor are you limited to just using a dedicated machine. You can setup the system as you see fit.
Why should I assemble a Cloud-Ripper?
Getting a Cloud-Ripper is not forced on anyone. You can put together whatever spare hardware you have (or just run it locally under Windows). Since the services are extremely lightweight, any x86 PC will do. If you invest in a ODroid N2 board ($80 range) then you can install all the services on that if you like. So if you have no interest in clustering or building your own supercomputer, then any PC, Laptop or IOT single-board computer(s) will do. Provided it yields more or equal power as the XU4 (!)
What you will experience with a dedicated cluster, regardless of putting the boards in a nice cube, is that you get excellent performance for very little money. It is quite amazing what $200 can buy you in 2019. And when you daisy chain 5 ODroid XU4 boards together on a switch, those 5 cheap boards will deliver the same serving power as an x86 setup costing twice as much.
The NVidia Jetson Nano SBC, one of the fastest boards available at under $100
Pico is offering 3 different packages. The most expensive option is the pre-assembled cube. This is for some reason priced at $750 which is completely absurd. If you can operate a screwdriver, then you can assemble the cube yourself in less than an hour. So the starter-kit case which costs $259 is more than enough.
Next, you can buy the XU4 boards directly from Hardkernel for $40 a piece, which will set you back $200. If you order the Pico 5H case as a kit, that brings the sub-total up to $459. But that price-tag includes everything you need except sd-cards. So the kit contains power-supply, the electrical wiring, a fast gigabit ethernet switch [built-into the cube], active cooling, network cables and power cables. You don’t need more than 8Gb sd-cards, which costs practically nothing these days.
Note: The Quartex Media Desktop “file-service” should have a dedicated disk. I bought a 256Gb SSD disk with a USB 3.0 interface, but you can just use a vanilla USB stick to store user-account data + user files.
As a bonus, such a setup is easy to recycle should you want to do something else later. Perhaps you want to learn more about Kubernetes? What about a docker-swarm? A freepascal build-server perhaps? Why not install FreeNas, Plex, and a good backup solution? You can set this up as you can afford. If 5 x ODroid XU4 is too much, then get 3 of them instead + the Pico 3H case.
So should Quartex Media Desktop not be for you, or you want to do something else entirely — then having 5 ODroid XU4 boards around the house is not a bad thing.
Oh and if you want some serious firepower, then order the Pico 5H kit for the NVidia Jetson Nano boards. Graphically those boards are beyond any other SoC on the market (in it’s price range). But as a consequence the Jetson Nano starts at $99. So for a full kit you will end up with $500 for the boards alone. But man those are the proverbial Ferrari of IOT.
Quartex Media Desktop, new compiler and general progress
It’s been a few weeks since my last update on the project. The reason I dont blog that often about Quartex Media Desktop (QTXMD), is because the official user-group has grown to 2000+ members. So it’s easier for me to post developer updates directly to the audience rather than writing articles about it.
Quartex Media Desktop ~ a complete environment that runs on every device
If you haven’t bothered digging into the project, let me try to sum it up for you quickly.
Quick recap on Quartex Media Desktop
To understand what makes this project special, first consider the relationship between Microsoft Windows and a desktop program. The operating system, be it Windows, Linux or OSX – provides an infrastructure that makes complex applications possible. The operating-system offers functions and services that programs can rely on.
The most obvious being:
- A filesystem and the ability to save and load data
- A windowing toolkit so programs can be displayed and have a UI
- A message system so programs can communicate with the OS
- A service stack that takes care of background tasks
- Authorization and identity management (security)
I have just described what the Quartex Media Desktop is all about. The goal is simple:
to provide for JavaScript what Windows and OS X provides for ordinary programs.
Just stop and think about this. Every “web application” you have ever seen, have all lacked these fundamental features. Sure you have libraries that gives you a windowing environment for Javascript, like Embarcadero Sencha; but im talking about something a bit more elaborate. Creating windows and buttons is easy, but what about ownership? A runtime environment has to keep track of the resources a program allocates, and make sure that security applies at every step.
Target audience and purpose
Take a second and think about how many services you use that have a web interface. In your house you probably have a router, and all routers can be administered via the browser. Sadly, most routers operate with a crude design and that leaves much to be desired.
Router interfaces for web are typically very limited and plain looking. Imagine what NetGear could do with Quartex Media Desktop instead
If you like to watch movies you probably have a Plex or Kodi system running somewhere in your house; perhaps you access that directly via your TV – or via a modern media system like Playstation 4 or XBox one. Both Plex and Kodi have web-based interfaces.
Netflix is now omnipresent and have practically become an institution in it’s own right. Netflix is often installed as an app – but the app is just a thin wrapper around a web-interface. That way they dont have to code apps for every possible device and OS out there.
If you commute via train in Scandinavia, chances are you buy tickets on a kiosk booth. Most of these booths run embedded software and the interface is again web based. That way they can update the whole interface without manually installing new software on each device.
Plex is a much loved system. It is based on a mix of web and native technologies
These are just examples of web based interfaces you might know and use; devices that leverage web technology. As a developer, wouldn’t it be cool if there was a system that could be forked, adapted and provide advanced functionality out of the box?
Just imagine a cheap Jensen router with a Quartex Media Desktop interface! It could provide a proper UI interface with applications that run in a windowing environment. They could disable ordinary desktop functionality and run their single application in kiosk mode. Taking full advantage of the underlying functionality without loss of security.
And the same is true for you. If you have a great idea for a web based application, you can fork the system, adjust it to suit your needs – and deploy a cutting edge cloud system in days rather than months!
New compiler?
Up until recently I used Smart Mobile Studio. But since I have left that company, the matter became somewhat pressing. I mean, QTXMD is an open-source system and cant really rely on third-party intellectual property. Eventually I fired up Delphi, forked the latest DWScript, and used that to roll a new command-line compiler.
Web technology has reached a level of performance that rivals native applications. You can pretty much retire Photoshop in favour of web based applications these days
But with a new compiler I also need a new RTL. Thankfully I have been coding away on the new RTL for over a year, but there is still a lot of work to do. I essentially have to implement the same functionality from scratch.
There will be more info on the new compiler / codegen when its production ready.
Progress
If I was to list all the work I have done since my last post, this article would be a small book. But to sum up the good stuff:
- Authentication has been moved into it’s own service
- The core (the main server) now delegates login messages to said service
- We no longer rely on the Smart Pascal filesystem drivers, but use the raw node.js functions instead (faster)
- The desktop now use the Smart Theme engine. This means that we can style the desktop to whatever we like. The OS4 theme that was hardcoded will be moved into its own proper theme-file. This means the user can select between OS4, iOS, Android and Ubuntu styling. Creating your own theme-files is also possible. The Smart theme-engine will be replaced by a more elaborate system in QTX later
- Ragnarok (the message api) messages now supports routing. If a routing structure is provided, the core will relay the message to the process in question (providing security allows said routing for the user)
- The desktop now checks for .info files when listing a directory. If a file is accompanied by an .info file, the icon is extracted and shown for that file
- Most of the service layer now relies on the QTX RTL files. We still have some dependencies on the Smart Pascal RTL, but we are making good progress on QTX. Eventually the whole system will have no dependencies outside QTX – and can thus be compiled without any financial obligations.
- QTX has it’s own node.js classes, including server and client base-classes
- Http(s) client and server classes are added to QTX
- Websocket and WebSocket-Secure are added to QTX
- TQTXHybridServer unifies http and websocket. Meaning that this server type can handle both orinary http requests – but also websocket connections on the same network socket. This is highly efficient for websocket based services
- UDP classes for node.js are implemented, both client and server
- Zero-Config classes are now added. This is used by the core for service discovery. Meaning that the child services hosted on another machine will automatically locate the core without knowing the IP. This is very important for machine clustering (optional, you can define a clear IP in the core preferences file)
- Fixed a bug where the scrollbars would corrupt widget states
- Added API functions for setting the scrollbars from hosted applications (so applications can tell the desktop that it needs scrollbar, and set the values)
- .. and much, much more
I will keep you all posted about the progress — the core (the fundamental system) is set for release in december – so time is of the essence! Im allocating more or less all my free time to this, and it will be ready to rock around xmas.
When the core is out, I can focus solely on the applications. Everything from Notepad to Calculator needs to be there, and more importantly — the developer tools. The CloudForge IDE for developers is set for 2020. With that in place you can write applications for iOS, Android, Windows, OS X and Linux directly from Quartex Media Desktop. Nothing to install, you just need a modern browser and a QTX account.
The system is brilliant for small teams and companies. They can setup their own instance, communicate directly via the server (text chat and video chat is scheduled) and work on their products in concert.
Calling node.js from Delphi
We got a good question about how to start a node.js program from Delphi on our Facebook group today (third one in a week?). When you have been coding for years you often forget that things like this might not be immediately obvious. Hopefully I can shed some light on the options in this post.
Node or chrome?
Just to be clear: node.js has nothing to do with chrome or chromium embedded. Chrome is a web-browser, a completely visual environment and ecosystem.
Node.js is the complete opposite. It is purely a shell based environment, meaning that it’s designed to run services and servers, with emphasis on the latter.
The only thing node.js and chrome have in common, is that they both use the V8 JavaScript runtime engine to load, JIT compile and execute scripts at high speed. Beyond that, they are utterly alien to each other.
Can node.js be embedded into a Delphi program?
Technically there is nothing stopping a C/C++ developer from compiling the node.js core system as C++ builder compatible .obj files; files that can then be linked into a Delphi application through references. But this also requires a bit of scaffolding, like adding support for malloc_, free_ and a few other procedures – so that your .obj files uses the same memory manager as your Delphi code. But until someone does just that and publish it, im afraid you are stuck with two options:
- Use a library called Toby, that keeps node.js in a single DLL file. This is the most practical way if you insist on hosting your own version of node.js
- Add node.js as a prerequisite and give users the option to locate the node.exe in your application’s preferences. This is the way I would go, because you really don’t want to force users to stick with your potentially outdated or buggy build.
So yes, you can use toby and just add the toby dll file to your program folder, but I have to strongly advice against that. There is no point setting yourself up for maintaining a whole separate programming language, just because you want JavaScript support.
“How many in your company can write high quality WebAssembly modules?”
If all you want to do is support JavaScript in your application, then I would much rather install Besen into Delphi. Besen is a JavaScript runtime engine written in Freepascal. It is fully compatible with Delphi, and follows the ECMA standard to the letter. So it is extremely compatible, fast and easy to use.
Like all Delphi components Besen is compiled into your application, so you have no dependencies to worry about.
Starting a node.js script
The easiest way to start a node.js script, is to simply shell-execute out of your Delphi application. This can be done as easily as:
ShellExecute(Handle, 'open', PChar('node.exe'), pchar('script.js'), nil, SW_SHOW);
This is more than enough if you just want to start a service, server or do some work that doesn’t require that you capture the result.
If you need to capture the result, the data that your node.js program emits on stdout, there is a nice component in the Jedi Component Library. Also plenty of examples online on how to do that.
If you need even further communication, you need to look for a shell-execute that support pipes. All node.js programs have something called a message-channel in the Javascript world. In reality though, this is just a named pipe that is automatically created when your script starts (with the same moniker as the PID [process identifier]).
If you opt for the latter you have a direct, full duplex message channel directly into your node.js application. You just have to agree with yourself on a protocol so that your Delphi code understands what node.js is saying, and visa versa.
UDP or TCP
If you don’t want to get your hands dirty with named pipes and rolling your own protocol, you can just use UDP to let your Delphi application communicate with your node.js process. UDP is practically without cost since its fundamental to all networking stacks, and in your case you will be shipping messages purely between processes on localhost. Meaning: packets are never sent on the network, but rather delegated between processes on the same machine.
In that case, I suggest you ship in the port you want your UDP server to listen on, so that your node.js service acts as the server. A simple command-line statement like:
node.exe myservice.js 8090
Inside node.js you can setup an UDP server with very little fuzz:
function setupServer(port) {
var os = require("os");
var dgram = require("dgram");
var socket = dgram.createSocket("udp4");
var MULTICAST_HOST = "224.0.0.236";
var BROADCAST_HOST = "255.255.255.255";
var ALL_PORT = 60540;
var MULTICAST_TTL = 1; // Local network
socket.bind(port);
socket.on('listening', function() {
socket.setMulticastLoopback(true);
socket.setMulticastTTL(MULTICAST_TTL);
socket.addMembership(multicastHost);
if(broadcast) { socket.setBroadcast(true); }
});
socket.on('message', parseMessage);
}
function parseMessage(message, rinfo) {
try {
var messageObject = JSON.parse(message);
var eventType = messageObject.eventType;
} catch(e) {
}
}
Note: the code above assumes a JSON text message.
You can then use any Delphi UDP client to communicate with your node.js server, Indy is good, Synapse is a good library with less overhead – there are many options here.
Do I have to learn Javascript to use node.js?
If you download DWScript you can hook-up the JS-codegen library (see library folder in the DWScript repository), and use that to compile DWScript (object pascal) to kick-ass Javascript. This is the same compiler that was used in Smart Mobile Studio.
“Adding WebAssembly to your resume is going to be a hell of a lot more valuable in the years to come than C# or Java”
Another alternative is to use Freepascal, they have a pas2js project where you can compile ordinary object-pascal to javascript. Naturally there are a few things to keep in mind, both for DWScript and Freepascal – like avoiding pointers. But clean object pascal compiles just fine.
If JavaScript is not your cup of tea, or you simply don’t have time to learn the delicate nuances between the DOM (document object model, used by browsers) and the 100% package oriented approach deployed by node.js — then you can just straight up to webassembly.
RemObjects Software has a kick-ass webassembly compiler, perfect if you dont have the energy or time to learn JavaScript. As of writing this is the fastest and most powerful toolchain available. And I have tested them all.
WebAssembly, no Javascript needed
You might remember Oxygene? It used to be shipped with Delphi as a way to target Microsoft CLR (common language runtime) and the .net framework.
Since then Oxygene and the RemObjects toolchain has evolved dramatically and is now capable of a lot more than CLR support.
- You can compile to raw, llvm optimized machine code for 8 platforms
- You can compile to CLR/.Net
- You can compile to Java bytecodes
- You can compile to WebAssembly!
WebAssembly is not Javascript, it’s important to underline that. WebAssembly was created especially for developers using traditional languages, so that traditional compilers can emit web friendly, binary code. Unlike Javascript, WebAssembly is a purely binary format. Just like Delphi generates machine-code that is linked into a final executable, WebAssembly is likewise compiled, linked and emitted in binary form.
If that sounds like a sales pitch, it’s not. It’s a matter of practicality.
- WebAssembly is completely barren out of the box. The runtime environment, be it V8 for the browser or V8 for node.js, gives you nothing out of the box. You don’t even have WriteLn() to emit text.
- Google expects compiler makers to provide their own RTL functions, from the fundamental to the advanced. The only thing V8 gives you, is a barebone way of referencing objects and functions on the other side, meaning the JS and DOM world. And that’s it.
So the reason i’m talking a lot about Oxygene and RemObjects Elements (Elements is the name of the compiler toolchain RemObjects offers), is because it ships with an RTL. So you are not forced to start on actual, literal assembly level.
If you don’t want to study JavaScript, Oxygene and Elements from RemObjects is the solution
RemObjects also delivers a DelphiVCL compatibility framework. This is a clone of the Delphi VCL / Freepascal LCL. Since WebAssembly is still brand new, work is being done on this framework on a daily basis, with updates being issued all the time.
Note: The Delphi VCL framework is not just for WebAssembly. It represents a unified framework that can work anywhere. So if you switch from WebAssembly to say Android, you get the same result.
The most important part of the above, is actually not the visual stuff. I mean, having HTML5 visual controls is cool – but chances are you want to use a library like Sencha, SwiftUI or jQueryUI to compose your forms right? Which means you just want to interface with the widgets in the DOM to set and get values.
You probably want to use a fancy UI library, like jQuery UI. This works perfectly with Elements because you can reference the controls from your WebAssembly module. You dont have to create TButton, TListbox etc manually
The more interesting stuff is actually the non-visual code you get access to. Hundreds of familiar classes from the VCL, painstakingly re-created, and usable from any of the 5 languages Elements supports.
You can check it out here: https://github.com/remobjects/DelphiRTL
Skipping JavaScript all together
I dont believe in single languages. Not any more. There was a time when all you needed was Delphi and a diploma and you were set to conquer the world. But those days are long gone, and a programmer needs to be flexible and have a well stocked toolbox.
Knowing where you want to be is half the journey
The world really don’t need yet-another-c# developer. There are millions of C# developers in India alone. C# is just “so what?”. Which is also why C# jobs pays less than Delphi or node.js system service jobs.
What you want, is to learn the things others avoid. If JavaScript looks alien and you feel uneasy about the whole thing – that means you are growing as a developer. All new things are learned by venturing outside your comfort zone.
How many in your company can write high quality WebAssembly modules?
How many within one hour driving distance from your office or home are experts at WebAssembly? How many are capable of writing industrial scale, production ready system services for node.js that can scale from a single instance to 1000 instances in a large, clustered cloud environment?
Any idiot can pick up node.js and knock out a service, but with your background from Delphi or C++ builder you have a massive advantage. All those places that can throw an exception that JS devs usually ignore? As a Delphi or Oxygene developer you know better. And when you re-apply that experience under a different language, suddenly you can do stuff others cant. Which makes your skills valuable.
The Quartex Media Desktop have made even experienced node / web developers gasp. They are not used to writing custom-controls and large-scale systems, which is my advantage
So would you learn JavaScript or just skip to WebAssembly? Honestly? Learn a bit of both. You don’t have to be an expert in JavaScript to compliment WebAssembly. Just get a cheap book, like “Node.js for beginners” and “JavaScript the good parts” ($20 a piece) and that should be more than enough to cover the JS side of things.
Adding WebAssembly to your resume and having the material to prove you know your stuff, is going to be a hell of a lot more valuable in the years to come than C#, Java or Python. THAT I can guarantee you.
And, we have a wicked cool group on Facebook you can join too: Click here to visit RemObjects Developer.
Enumerating network adapters in DWScript/Smart under Node.js
This is something I never had the time to implement under Smart Pascal, but it should be easy enough to patch. If you are using DWScript with the QTX Framework this is already in place. But for Smart users, here is a quick recipe.
First, we need access to the node.js OS module:
unit qtx.node.os;
//#############################################################################
// Quartex RTL for DWScript
// Written by Jon L. Aasenden, all rights reserved
// This code is released under modified LGPL (see license.txt)
//#############################################################################
unit NodeJS.os;
interface
uses
NodeJS.Core;
type
TCpusResultObjectTimes = class external
property user: Integer;
property nice: Integer;
property sys: Integer;
property idle: Integer;
property irq: Integer;
end;
TCpusResult = class external
property model: String;
property speed: Integer;
property times: TcpusResultObjectTimes;
end;
JNetworkInterfaceInfo = class external
property address: string;
property netmask: string;
property family: string;
property mac: string;
property scopeid: integer;
property internal: boolean;
property cidr: string;
end;
Jos_Exports = class external
public
function tmpDir: String;
function hostname: String;
function &type: String;
function platform: String;
function arch: String;
function release: String;
function uptime: Integer;
function loadavg: array of Integer;
function totalmem: Integer;
function freemem: Integer;
function cpus: array of TCpusResult;
function networkInterfaces: variant;
property EOL: String;
end;
function NodeJSOsAPI: Jos_Exports;
implementation
function NodeJSOsAPI: Jos_Exports;
begin
result := Jos_Exports(RequireModule("os") );
end;
end.
With that in place, we can start enumerating through the adapters. Remember that a PC can have several adapters attached, from a dedicated card to X number of USB wifi sticks.
Here is a little routine that goes through the adapters, and returns the first IPv4 LAN address it finds. This is very useful when writing servers, since you need the IP + port to setup a binding. And yes, you can just call HostName(), but the point here is to know how to run through the adapter array.
function GetMyV4LanIP: string;
begin
var OSAPI := NodeJSOsAPI();
var NetAdapters := OSAPI.networkInterfaces();
for var Adapter in NetAdapters do
begin
// Skip loopback device
if Adapter.Contains('Loopback') then
continue;
for var netIntf in NetAdapters[Adapter] do
begin
var address = JNetworkInterfaceInfo( NetAdapters[Adapter][netIntf] );
if not address.internal then
begin
// force copy of string
var lFam: string := string(address.family) + " ";
// make sure its ipv4
if lFam.ToLower().Trim() = 'ipv4' then
begin
result := address.address + " ";
result := result.trim();
break;
end;
end;
end;
end;
if result.length < 1 then
result := '127.0.0.1';
end;
Porting TextCraft to Oxygene
TextCraft is a simple yet powerful text parser, designed for general purpose parsing jobs. I originally implemented it for Delphi, it’s the base-parser for the LDEF bytecode assembler amongst other things. It was ported to Smart Pascal, then Freepascal – and now finally Oxygene.
The LDEF Assembler is a part of the Quartex Media Desktop
The LDEF assembler and bytecode engine is currently implemented in Smart and compiles for Javascript. It’s a complete assembler and VM allowing coders to approach Asm.js from an established instruction-set. In short: you feed it source-code, it spits out bytecodes that you can execute super fast in either the browser or elsewhere. As long as there is a VM implementation available.
The Javascript version works really well, especially on node.js. In essence, i don’t need to re-compile the toolchain when moving between arm, x86, windows, linux or osx. Think of it as a type of Java bytecodes or CLR bytecodes.
Getting the code to run under Oxygene, means that I can move the whole engine into WebAssembly. The parser, assembler and linker (et-al) can thus run as WebAssembly, and I can use that from my JavaScript front-end code. Best of both worlds – the flamboyant creativity of JavaScript, and the raw speed of WebAssembly.
The port
Before I can move over the top-level parser + assembler etc, the generic parser code has to work. I was reluctant to start because I imagined the porting would take at least a day, but luckily it took me less than an hour. There are a few superficial differences between Smart, Delphi, Freepascal and Oxygene; for example the Copy() function for strings is not a lose function in Oxygene, instead you use String.SubString(). Functions like High() and Low() on strings likewise has to be refactored.
But all in all the conversion was straight-forward, and TextCraft is now a part of the QTX library for Oxygene. I’ll be uploading a commit to GIT with the whole shabam soon.
Well, hope the WordPress parser doesnt screw this up too bad.
namespace qtxlib;
//##################################################################
// TextCraft 1.2
// Written by Jon L. Aasenden
//
// This is a port of TC 1.2 from Freepascal. TextCraft is initially
// a Delphi parser framework. The original repository can be found
// on BitBucket at:
//
// https://bitbucket.org/hexmonks/main
//
//##################################################################
{$DEFINE USE_INCLUSIVE}
{$define USE_BMARK}
interface
uses
qtxlib, System, rtl,
RemObjects.Elements.RTL.Delphi,
RemObjects.Elements.RTL.Delphi.VCL;
type
// forward declarations
TTextBuffer = class;
TParserContext = class;
TCustomParser = class;
TParserModelObject = class;
// Exceptions
ETextBuffer = class(Exception);
EModelObject = class(Exception);
// Callback functions
TTextValidCB = function (Item: Char): Boolean;
// Bookmark datatype
TTextBufferBookmark = class
public
property bbOffset: Integer;
property bbCol: Integer;
property bbRow: Integer;
function Equals(const ThisMark: TTextBufferBookmark): Boolean;
end;
{.$DEFINE USE_BMARK}
TTextBuffer = class(TErrorObject)
private
FData: String;
FOffset: Integer;
FLength: Integer;
FCol: Integer;
FRow: Integer;
{$IFDEF USE_BMARK}
FBookmarks: List;
{$ENDIF}
procedure SetCacheData(NewText: String);
public
property Column: Integer read FCol;
property Row: Integer read FRow;
property Count: Integer read FLength;
property Offset: Integer read FOffset;
property CacheData: String read FData write SetCacheData;
// These functions map directly to the "Current"
// character where the offset is placed, and is used to
// write code that makes more sense to human eyes
function CrLf: Boolean;
function Space: Boolean;
function Tab: Boolean;
function SemiColon: Boolean;
function Colon: Boolean;
function ConditionEnter: Boolean;
function ConditionLeave: Boolean;
function BracketEnter: Boolean;
function BracketLeave: Boolean;
function Ptr: Boolean;
function Punctum: Boolean;
function Question: Boolean;
function Less: Boolean;
function More: Boolean;
function Equal: Boolean;
function Pipe: Boolean;
function Numeric: Boolean;
function Empty: Boolean;
function BOF: Boolean;
function EOF: Boolean;
function Current: Char;
function First: Boolean;
function Last: Boolean;
// Same as "Next", but does not automatically
// consume CR+LF, used when parsing textfragments
function NextNoCrLf: Boolean;
// Normal Next function, will automatically consume
// CRLF when it encounters it
function Next: Boolean;
function Back: Boolean;
function Bookmark: TTextBufferBookmark;
procedure Restore(const Mark: TTextBufferBookmark);
{$IFDEF USE_BMARK}
procedure Drop;
{$ENDIF}
procedure ConsumeJunk;
procedure ConsumeCRLF;
function Compare(const CompareText: String;
const CaseSensitive: Boolean): Boolean;
function Read(var Fragment: Char): Boolean; overload;
function Read: Char; overload;
function ReadTo(const CB: TTextValidCB; var TextRead: String): Boolean; overload;
function ReadTo(const Resignators: TSysCharSet; var TextRead: String): Boolean; overload;
function ReadTo(MatchText: String): Boolean; overload;
function ReadTo(MatchText: String; var TextRead: String): Boolean; overload;
function ReadToEOL: Boolean; overload;
function ReadToEOL(var TextRead: String): Boolean; overload;
function Peek: Char; overload;
function Peek(CharCount: Integer; var TextRead: String): Boolean; overload;
function NextNonControlChar(const CompareWith: Char): Boolean;
function NextNonControlText(const CompareWith: String): Boolean;
function ReadWord(var TextRead: String): Boolean;
function ReadQuotedString: String;
function ReadCommaList(var cList: List): Boolean;
function NextLine: Boolean;
procedure Inject(const TextToInject: String);
function GetCurrentLocation: TTextBufferBookmark;
function Trail: String;
procedure Clear;
procedure LoadBufferText(const NewBuffer: String);
constructor Create(const BufferText: String); overload; virtual;
finalizer;
begin
{$IFDEF USE_BMARK}
FBookmarks.Clear();
disposeAndNil(FBookmarks);
{$endif}
Clear();
end;
end;
TParserContext = class(TErrorObject)
private
FBuffer: TTextBuffer;
FStack: Stack;
public
property Buffer: TTextBuffer read FBuffer;
property Model: TParserModelObject;
procedure Push(const ModelObj: TParserModelObject);
function Pop: TParserModelObject;
function Peek: TParserModelObject;
procedure ClearStack;
constructor Create(const SourceCode: String); reintroduce; virtual;
finalizer;
begin
FStack.Clear();
FBuffer.Clear();
disposeAndNil(FStack);
disposeAndNil(FBuffer);
end;
end;
TCustomParser = class(TErrorObject)
private
FContext: TParserContext;
protected
procedure SetContext(const NewContext: TParserContext);
public
property Context: TParserContext read FContext;
function Parse: Boolean; virtual;
constructor Create(const ParseContext: TParserContext); reintroduce; virtual;
end;
TParserModelObject = class(TObject)
private
FParent: TParserModelObject;
FChildren: List;
protected
function GetParent: TParserModelObject; virtual;
function ChildGetCount: Integer; virtual;
function ChildGetItem(const Index: Integer): TParserModelObject; virtual;
function ChildAdd(const Instance: TParserModelObject): TParserModelObject; virtual;
public
property Parent: TParserModelObject read GetParent;
property Context: TParserContext;
procedure Clear; virtual;
constructor Create(const AParent: TParserModelObject); virtual;
finalizer;
begin
Clear();
FChildren := nil;
end;
end;
implementation
//#####################################################################
// Error messages
//#####################################################################
const
CNT_ERR_BUFFER_EMPTY = 'Buffer is empty error';
CNT_ERR_OFFSET_BOF = 'Offset at BOF error';
CNT_ERR_OFFSET_EOF = 'Offset at EOF error';
CNT_ERR_COMMENT_NOTCLOSED = 'Comment not closed error';
CNT_ERR_OFFSET_EXPECTED_EOF = 'Expected EOF error';
CNT_ERR_LENGTH_INVALID = 'Invalid length error';
//#####################################################################
// TTextBufferBookmark
//#####################################################################
function TTextBufferBookmark.Equals(const ThisMark: TTextBufferBookmark): boolean;
begin
result := ( (ThisMark nil) and (ThisMark self) )
and (self.bbOffset = ThisMark.bbOffset)
and (self.bbCol = ThisMark.bbCol)
and (self.bbRow = ThisMark.bbRow);
end;
//#####################################################################
// TTextBuffer
//#####################################################################
constructor TTextBuffer.Create(const BufferText: string);
begin
inherited Create();
if length(BufferText) > 0 then
LoadBufferText(BufferText)
else
Clear();
end;
procedure TTextBuffer.Clear;
begin
FData := '';
FOffset := -1;
FLength := 0;
FCol := -1;
FRow := -1;
{$IFDEF USE_BMARK}
FBookmarks.Clear();
{$ENDIF}
end;
procedure TTextBuffer.SetCacheData(NewText: string);
begin
LoadBufferText(NewText);
end;
function TTextBuffer.Trail: string;
begin
if not Empty then
begin
if not EOF then
result := FData.Substring(FOffset, length(FData) );
//result := Copy( FData, FOffset, length(FData) );
end;
end;
procedure TTextBuffer.LoadBufferText(const NewBuffer: string);
begin
// Flush existing buffer
Clear();
// Load in buffertext, init offset and values
var TempLen := NewBuffer.Length;
if TempLen > 0 then
begin
FData := NewBuffer;
FOffset := 0; // start at BOF
FCol := 0;
FRow := 0;
FLength := TempLen;
end;
end;
function TTextBuffer.GetCurrentLocation: TTextBufferBookmark;
begin
if Failed then
ClearLastError();
if not Empty then
begin
result := TTextBufferBookmark.Create;
result.bbOffset := FOffset;
result.bbCol := FCol;
result.bbRow := FRow;
end else
raise ETextBuffer.Create
('Failed to return position, buffer is empty error');
end;
function TTextBuffer.Bookmark: TTextBufferBookmark;
begin
if Failed then
ClearLastError();
if not Empty then
begin
result := TTextBufferBookmark.Create;
result.bbOffset := FOffset;
result.bbCol := FCol;
result.bbRow := FRow;
{$IFDEF USE_BMARK}
FBookmarks.add(result);
{$ENDIF}
end else
raise ETextBuffer.Create
('Failed to bookmark location, buffer is empty error');
end;
procedure TTextBuffer.Restore(const Mark: TTextBufferBookmark);
begin
if Failed then
ClearLastError();
if not Empty then
begin
if Mark nil then
begin
FOffset := Mark.bbOffset;
FCol := Mark.bbCol;
FRow := Mark.bbRow;
Mark.Free;
{$IFDEF USE_BMARK}
var idx := FBookmarks.Count;
if idx > 0 then
begin
dec(idx);
FOffset := FBookmarks[idx].bbOffset;
FCol := FBookmarks[idx].bbCol;
FRow := FBookmarks[idx].bbRow;
FBookmarks.Remove(idx);
//FBookmarks.SetLength(idx)
//FBookmarks.Delete(idx,1);
end else
raise ETextBuffer.Create('Failed to restore bookmark, none exist');
{$ENDIF}
end else
raise ETextBuffer.Create('Failed to restore bookmark, object was nil error');
end else
raise ETextBuffer.Create
('Failed to restore bookmark, buffer is empty error');
end;
{$IFDEF USE_BMARK}
procedure TTextBuffer.Drop;
begin
if Failed then
ClearLastError();
if not Empty then
begin
if FBookmarks.Count > 0 then
FBookmarks.Remove(FBookmarks.Count-1)
else
raise ETextBuffer.Create('Failed to drop bookmark, none exist');
end else
raise ETextBuffer.Create
('Failed to drop bookmark, buffer is empty error');
end;
{$ENDIF}
function TTextBuffer.Read(var Fragment: char): boolean;
begin
if Failed then
ClearLastError();
if not Empty then
begin
result := FOffset <= length(FData);
if result then
begin
// return character
Fragment := FData[FOffset];
// update offset
inc(FOffset)
end else
begin
// return invalid char
Fragment := #0;
// Set error reason
SetLastError('Offset at BOF error');
end;
end else
begin
result := false;
Fragment := #0;
SetLastError('Buffer is empty error');
end;
end;
function TTextBuffer.Read: char;
begin
if Failed then
ClearLastError();
if not Empty then
begin
result := Current;
Next();
end else
result := #0;
end;
function TTextBuffer.ReadToEOL: boolean;
begin
if Failed then
ClearLastError();
if not Empty() then
begin
if BOF() then
begin
if not First() then
exit;
end;
if EOF() then
begin
SetLastError(CNT_ERR_OFFSET_EOF);
exit;
end;
// Keep start
var LStart := FOffset;
// Enum until match of EOF
{$IFDEF USE_INCLUSIVE}
repeat
if (FData[FOffset] = #13)
and (FData[FOffset + 1] = #10) then
begin
result := true;
break;
end else
begin
inc(FOffset);
inc(FCol);
end;
until EOF();
{$ELSE}
While FOffset < High(FData) do
begin
if (FData[FOffset] = #13)
and (FData[FOffset + 1] = #10) then
begin
result := true;
break;
end else
begin
inc(FOffset);
inc(FCol);
end;
end;
{$ENDIF}
// Last line in textfile might not have
// a CR+LF, so we have to check for termination
if not result then
begin
if EOF then
begin
if LStart = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset <= high(FData) ) )
and ( (FData[FOffset] = '= Low(FData)) and (FOffset ') );
end;
function TTextBuffer.Equal: boolean;
begin
result := (not Empty)
and ( (FOffset >= Low(FData)) and (FOffset = Low(FData)) and (FOffset = Low(FData)) and (FOffset LStart then
begin
// Any text to return? Or did we start
// directly on a CR+LF and have no text to give?
var LLen := FOffset - LStart;
TextRead := FData.Substring(LStart, LLen);
//TextRead := Copy(FData, LStart, LLen);
end;
// Either way, we exit because CR+LF has been found
result := true;
break;
end;
inc(FOffset);
inc(FCol);
until EOF();
{$ELSE}
While FOffset LStart then
begin
// Any text to return? Or did we start
// directly on a CR+LF and have no text to give?
var LLen := FOffset - LStart;
TextRead := copy(FData, LStart, LLen);
end;
// Either way, we exit because CR+LF has been found
result := true;
break;
end;
inc(FOffset);
inc(FCol);
end;
{$ENDIF}
// Last line in textfile might not have
// a CR+LF, so we have to check for EOF and treat
// that as a terminator.
if not result then
begin
if FOffset >= high(FData) then
begin
if LStart 0 then
begin
TextRead := FData.Substring(LStart, LLen);
//TextRead := Copy(FData, LStart, LLen);
result := true;
end;
exit;
end;
end;
end;
end;
end;
function TTextBuffer.ReadTo(const CB: TTextValidCB; var TextRead: string): boolean;
begin
if Failed then
ClearLastError();
TextRead := '';
if not Empty then
begin
if BOF() then
begin
if not First() then
exit;
end;
if EOF() then
begin
SetLastError(CNT_ERR_OFFSET_EOF);
exit;
end;
if not assigned(CB) then
begin
SetLastError('Invalid callback handler');
exit;
end;
{$IFDEF USE_INCLUSIVE}
repeat
if not CB(Current) then
break
else
TextRead := TextRead + Current;
if not Next() then
break;
until EOF();
{$ELSE}
while not EOF do
begin
if not CB(Current) then
break
else
TextRead := TextRead + Current;
if not Next() then
break;
end;
{$ENDIF}
result := TextRead.Length > 0;
end else
begin
result := false;
SetLastError(CNT_ERR_BUFFER_EMPTY);
end;
end;
function TTextBuffer.ReadTo(const Resignators: TSysCharSet; var TextRead: string): boolean;
begin
if Failed then
ClearLastError();
TextRead := '';
if not Empty then
begin
if BOF() then
begin
if not First() then
exit;
end;
if EOF() then
begin
SetLastError(CNT_ERR_OFFSET_EOF);
exit;
end;
{$IFDEF USE_INCLUSIVE}
repeat
if not Resignators.Contains(Current) then
TextRead := TextRead + Current
else
break;
if not Next() then
break;
until EOF();
{$ELSE}
while not EOF do
begin
if not (Current in Resignators) then
TextRead := TextRead + Current
else
break;
if not Next() then
break;
end;
{$ENDIF}
result := TextRead.Length > 0;
end else
begin
result := false;
SetLastError(CNT_ERR_BUFFER_EMPTY);
end;
end;
function TTextBuffer.ReadTo(MatchText: string): boolean;
begin
if Failed then
ClearLastError();
if not Empty() then
begin
if BOF() then
begin
if not First() then
exit;
end;
if EOF() then
begin
SetLastError(CNT_ERR_OFFSET_EOF);
exit;
end;
var MatchLen := length(MatchText);
if MatchLen > 0 then
begin
MatchText := MatchText.ToLower();
repeat
var TempCache := '';
if Peek(MatchLen, TempCache) then
begin
TempCache := TempCache.ToLower();
result := SameText(TempCache, MatchText);
if result then
break;
end;
if not Next then
break;
until EOF;
end;
end else
begin
result := false;
SetLastError(CNT_ERR_BUFFER_EMPTY);
end;
end;
function TTextBuffer.ReadTo(MatchText: string; var TextRead: string): boolean;
begin
if Failed then
ClearLastError();
result := false;
TextRead := '';
if not Empty() then
begin
if BOF() then
begin
if not First() then
exit;
end;
if EOF() then
begin
SetLastError(CNT_ERR_OFFSET_EOF);
exit;
end;
if MatchText.Length > 0 then
begin
MatchText := MatchText.ToLower();
repeat
var TempCache := '';
if Peek(MatchText.Length, TempCache) then
begin
TempCache := TempCache.ToLower();
result := SameText(TempCache, MatchText);
if result then
break
else
TextRead := TextRead + Current;
end else
TextRead := TextRead + Current;
if not Next() then
break;
until EOF;
end;
end else
begin
result := false;
SetLastError(CNT_ERR_BUFFER_EMPTY);
end;
end;
procedure TTextBuffer.Inject(const TextToInject: string);
begin
if length(FData) > 0 then
begin
var lSeg1 := FData.Substring(1, FOffset);
var lSeg2 := FData.Substring(FOffset + 1, length(FData));
//var LSeg1 := Copy(FData, 1, FOffset);
//var LSeg2 := Copy(FData, FOffset+1, FData.Length);
FData := lSeg1 + TextToInject + lSeg2;
end else
FData := TextToInject;
end;
function TTextBuffer.Compare(const CompareText: string;
const CaseSensitive: boolean): boolean;
begin
if Failed then
ClearLastError();
if not Empty() then
begin
if BOF() then
begin
if not First() then
exit;
end;
if EOF() then
begin
SetLastError(CNT_ERR_OFFSET_EOF);
exit;
end;
var LenToRead := CompareText.Length;
if LenToRead > 0 then
begin
// Peek will set an error message if it
// fails, so we dont need to set anything here
var ReadData := '';
if Peek(LenToRead, ReadData) then
begin
case CaseSensitive of
false: result := ReadData.ToLower() = CompareText.ToLower();
true: result := ReadData = CompareText;
end;
end;
end else
SetLastError(CNT_ERR_LENGTH_INVALID);
end else
SetLastError(CNT_ERR_BUFFER_EMPTY);
end;
procedure TTextBuffer.ConsumeJunk;
begin
if Failed then
ClearLastError();
if not Empty then
begin
if BOF() then
begin
if not First() then
exit;
end;
if EOF() then
begin
SetLastError(CNT_ERR_OFFSET_EOF);
exit;
end;
repeat
case Current of
' ':
begin
end;
'"':
begin
break;
end;
#8, #09:
begin
end;
'/':
begin
(* Skip C style remark *)
if Compare('/*', false) then
begin
if ReadTo('*/') then
begin
inc(FOffset, 2);
Continue;
end else
SetLastError(CNT_ERR_COMMENT_NOTCLOSED);
end else
begin
(* Skip Pascal style remark *)
if Compare('//', false) then
begin
if ReadToEOL() then
begin
continue;
end else
SetLastError(CNT_ERR_OFFSET_EXPECTED_EOF);
end;
end;
end;
'(':
begin
(* Skip pascal style remark *)
if Compare('(*', false)
and not Compare('(*)', false) then
begin
if ReadTo('*)') then
begin
inc(FOffset, 2);
continue;
end else
SetLastError(CNT_ERR_COMMENT_NOTCLOSED);
end else
break;
end;
#13:
begin
if FData[FOffset + 1] = #10 then
inc(FOffset, 2)
else
inc(FOffset, 1);
//if Peek = #10 then
// ConsumeCRLF;
continue;
end;
#10:
begin
inc(FOffset);
continue;
end;
else
break;
end;
if not Next() then
break;
until EOF;
end else
SetLastError(CNT_ERR_BUFFER_EMPTY);
end;
procedure TTextBuffer.ConsumeCRLF;
begin
if not Empty then
begin
if BOF() then
begin
if not First() then
exit;
end;
if EOF() then
begin
SetLastError(CNT_ERR_OFFSET_EOF);
exit;
end;
if (FData[FOffset] = #13) then
begin
if FData[FOffset + 1] = #10 then
inc(FOffset, 2)
else
inc(FOffset);
inc(FRow);
FCol := 0;
end;
end;
end;
function TTextBuffer.Empty: boolean;
begin
result := FLength < 1;
end;
// This method will look ahead, skipping space, tab and crlf (also known
// as control characters), and when a non control character is found it will
// perform a string compare. This method uses a bookmark and will restore
// the offset to the same position as when it was entered.
//
// Notes: The method "NextNonControlChar" is a similar method that
// performs a char-only compare.
function TTextBuffer.NextNonControlText(const CompareWith: string): boolean;
begin
if Failed then
ClearLastError();
if not Empty then
begin
if BOF() then
begin
if not First() then
exit;
end;
if EOF() then
begin
SetLastError(CNT_ERR_OFFSET_EOF);
exit;
end;
var Mark := Bookmark();
try
// Iterate ahead
repeat
if not (Current in [' ', #13, #10, #09]) then
break;
Next();
until EOF();
// Compare unless we hit the end of the line
if not EOF then
result := Compare(CompareWith, false);
finally
Restore(Mark);
end;
end else
SetLastError(CNT_ERR_BUFFER_EMPTY);
end;
// This method will look ahead, skipping space, tab and crlf (also known
// as control characters), and when a non control character is found it will
// perform a string compare. This method uses a bookmark and will restore
// the offset to the same position as when it was entered.
function TTextBuffer.NextNonControlChar(const CompareWith: char): boolean;
begin
if Failed then
ClearLastError();
if not Empty then
begin
if BOF() then
begin
if not First() then
exit;
end;
if EOF() then
begin
SetLastError(CNT_ERR_OFFSET_EOF);
exit;
end;
var Mark := Bookmark();
try
repeat
if not (Current in [' ', #13, #10, #09]) then
break;
Next();
until EOF();
//if not EOF then
result := Current.ToLower() = CompareWith.ToLower();
//result := LowerCase(Current) = LowerCase(CompareWith);
finally
Restore(Mark);
end;
end else
SetLastError(CNT_ERR_BUFFER_EMPTY);
end;
function TTextBuffer.Peek: char;
begin
if Failed then
ClearLastError();
if not Empty then
begin
if (FOffset 0 do
begin
TextRead := TextRead + Current;
if not Next() then
break;
dec(CharCount);
end;
finally
Restore(Mark);
end;
result := TextRead.Length > 0;
end else
SetLastError(CNT_ERR_OFFSET_EOF);
end else
SetLastError(CNT_ERR_BUFFER_EMPTY);
end;
function TTextBuffer.First: boolean;
begin
if Failed then
ClearLastError();
if not Empty then
begin
FOffset := Low(FData);
result := true;
end else
SetLastError(CNT_ERR_BUFFER_EMPTY);
end;
function TTextBuffer.Last: boolean;
begin
if Failed then
ClearLastError();
if not Empty then
begin
FOffset := high(FData);
result := true;
end else
SetLastError(CNT_ERR_BUFFER_EMPTY);
end;
function TTextBuffer.NextNoCrLf: boolean;
begin
if Failed then
ClearLastError();
if not Empty then
begin
// Check that we are not EOF
result := FOffset <= high(FData);
if result then
begin
// Update offset into buffer
inc(FOffset);
// update column, but not if its in a lineshift
if not (FData[FOffset] in [#13, #10]) then
inc(FCol);
end else
SetLastError(CNT_ERR_OFFSET_EOF);
end else
SetLastError(CNT_ERR_BUFFER_EMPTY);
end;
function TTextBuffer.Next: boolean;
begin
if Failed then
ClearLastError();
if not Empty() then
begin
if BOF() then
begin
if not First() then
exit;
end;
if EOF() then
begin
SetLastError(CNT_ERR_OFFSET_EOF);
exit;
end;
// Update offset into buffer
inc(FOffset);
// update column
inc(FCol);
// This is the same as ConsumeCRLF
// But this does not generate any errors since we PEEK
// ahead into the buffer to make sure the combination
// is correct before we adjust the ROW + offset
if FOffset Low(FData));
if result then
dec(FOffset)
else
SetLastError(CNT_ERR_OFFSET_BOF);
end else
SetLastError(CNT_ERR_BUFFER_EMPTY);
end;
function TTextBuffer.Current: char;
begin
if Failed then
ClearLastError();
// Check that buffer is not empty
if not Empty then
begin
// Check that we are on char 1 or more
if FOffset >= Low(FData) then
begin
// Check that we are before or on the last char
if (FOffset <= high(FData)) then
result := FData[FOffset]
else
begin
SetLastError(CNT_ERR_OFFSET_EOF);
result := #0;
end;
end else
begin
SetLastError(CNT_ERR_OFFSET_BOF);
result := #0;
end;
end else
begin
SetLastError(CNT_ERR_BUFFER_EMPTY);
result := #0;
end;
end;
function TTextBuffer.BOF: boolean;
begin
if not Empty then
result := FOffset high(FData);
end;
function TTextBuffer.NextLine: boolean;
begin
if Failed then
ClearLastError();
if not Empty then
begin
// Make sure we offset to a valid character
// in the buffer.
ConsumeJunk();
if not EOF then
begin
var ThisRow := self.FRow;
while Row = ThisRow do
begin
Next();
if EOF then
break;
end;
result := (Row ThisRow) and (not EOF);
end;
end;
end;
function TTextBuffer.ReadWord(var TextRead: string): boolean;
begin
if Failed then
ClearLastError();
TextRead := '';
if not Empty then
begin
// Make sure we offset to a valid character
// in the buffer.
ConsumeJunk();
// Not at the end of the file?
if not EOF then
begin
repeat
var el := Current;
if (el in
[ 'A'..'Z',
'a'..'z',
'0'..'9',
'_', '-' ]) then
TextRead := TextRead + el
else
break;
if not NextNoCrLf() then
break;
until EOF;
result := TextRead.Length > 0;
end else
SetLastError('Failed to read word, unexpected EOF');
end else
SetLastError('Failed to read word, buffer is empty error');
end;
function TTextBuffer.ReadCommaList(var cList: List): boolean;
var
LTemp: String;
LValue: String;
begin
if cList = nil then
cList := new List
else
cList.Clear();
if not Empty then
begin
ConsumeJunk();
While not EOF do
begin
case Current of
#09:
begin
// tab, just skip
end;
#13, #10:
begin
// CR+LF, consume and continue;
ConsumeCRLF();
end;
#0:
begin
// Unexpected EOL
break;
end;
';':
begin
//Perfectly sound ending
result := true;
break;
end;
'"':
begin
LValue := ReadQuotedString;
if LValue.Length > 0 then
begin
cList.add(LValue);
LValue := '';
end;
end;
',':
begin
LTemp := LTemp.Trim();
if LTemp.Length>0 then
begin
cList.add(LTemp);
LTemp := '';
end;
end;
else
begin
LTemp := LTemp + Current;
end;
end;
if not Next() then
break;
end;
if LTemp.Length > 0 then
cList.add(LTemp);
result := cList.Count > 0;
end;
end;
function TTextBuffer.ReadQuotedString: string;
begin
if not Empty then
begin
if not EOF then
begin
// Make sure we are on the " entry quote
if Current '"' then
begin
SetLastError('Failed to read quoted string, expected index on " character error');
exit;
end;
// Skip the entry char
if not NextNoCrLf() then
begin
SetLastError('Failed to skip initial " character error');
exit;
end;
while not EOF do
begin
// Read char from buffer
var TempChar := Current;
// Closing of string? Exit
if TempChar = '"' then
begin
if not NextNoCrLf then
SetLastError('failed to skip final " character in string error');
break;
end;
result := result + TempChar;
if not NextNoCrLf() then
break;
end;
end;
end;
end;
//##########################################################################
// TParserModelObject
//##########################################################################
constructor TParserModelObject.Create(const AParent:TParserModelObject);
begin
inherited Create;
FParent := AParent;
FChildren := new List;
end;
function TParserModelObject.GetParent:TParserModelObject;
begin
result := FParent;
end;
procedure TParserModelObject.Clear;
begin
FChildren.Clear();
end;
function TParserModelObject.ChildGetCount: integer;
begin
result := FChildren.Count;
end;
function TParserModelObject.ChildGetItem(const Index: integer): TParserModelObject;
begin
result := TParserModelObject(FChildren[Index]);
end;
function TParserModelObject.ChildAdd(const Instance: TParserModelObject): TParserModelObject;
begin
if FChildren.IndexOf(Instance) < 0 then
FChildren.add(Instance);
result := Instance;
end;
//###########################################################################
// TParserContext
//###########################################################################
constructor TParserContext.Create(const SourceCode: string);
begin
inherited Create;
FBuffer := TTextBuffer.Create(SourceCode);
FStack := new Stack;
end;
procedure TParserContext.Push(const ModelObj: TParserModelObject);
begin
if Failed then
ClearLastError();
try
FStack.Push(ModelObj);
except
on e: Exception do
SetLastError('Internal error:' + e.Message);
end;
end;
function TParserContext.Pop: TParserModelObject;
begin
if Failed then
ClearLastError();
try
result := FStack.Pop();
except
on e: Exception do
SetLastError('Internal error:' + e.Message);
end;
end;
function TParserContext.Peek: TParserModelObject;
begin
if Failed then
ClearLastError();
try
result := FStack.Peek();
except
on e: Exception do
SetLastError('Internal error:' + e.Message);
end;
end;
procedure TParserContext.ClearStack;
begin
if Failed then
ClearLastError();
try
FStack.Clear();
except
on e: Exception do
SetLastError('Internal error:' + e.Message);
end;
end;
//###########################################################################
// TCustomParser
//###########################################################################
constructor TCustomParser.Create(const ParseContext: TParserContext);
begin
inherited Create;
FContext := ParseContext;
end;
function TCustomParser.Parse: boolean;
begin
result := false;
SetLastErrorF('No parser implemented for class %s',[ClassName]);
end;
procedure TCustomParser.SetContext(const NewContext: TParserContext);
begin
FContext := NewContext;
end;
end.
New job, new office, new adventures
It’s been roughly 4 weeks since I posted a status report on Amibian.js. I normally keep people up-to-date on facebook (the “Amiga Disrupt” and also “Delphi Developer” groups). It’s been a very hectic month so I fully understand that people are asking. So let’s look at where the project is at and where we are on the time-line.
For those that might not know, I decided to leave Embarcadero a couple of months ago. I will be working out may before I move on. I wanted to write about that myself in a clean fashion, but sadly the news broke on Facebook prematurely.
Long story short, I have been very fortunate to work at Embarcadero. I am not leaving because there is anything wrong or something like that. I was hired as SC for the EMEA regions, which basically made me the support and presenter for most of europe, parts of asia and the middle east. It’s been a great adventure, but ultimately I had to admit that my passion is coding and community work. Sales is a very important part of any company, but it’s not really my cup of tea; my passion has always been research and development.
So, come first of June and I start in a new position at RemObjects. A company that has deep roots with Delphi and C++ builder users – and a company that continues to produce a wealth of high-quality, high-performance frameworks for Delphi and C++ builder. RemObjects also has a strong focus on modern languages, and have a strong portfolio of new and exciting compilers and languages to offer. The Oxygene compiler should be no stranger to Delphi developers, a powerful object-pascal dialect that can target a variety of platforms and chipsets.
Since compiler technology and run-time systems has been my main focus for well over a decade now, I feel RemObjects is a better match.
Quartex Components
Quartex Components has been an officially registered Norwegian company for a while now, so perhaps not news. What is news is that it’s now directly connected with the development of the Quartex Media Desktop (codename “Amibian.js”). While Amibian.js is an open source endeavour, there will be both free and commercial products running on top of that platform. I have written at length about Cloud Forge in the past, so I wont re-hash that again. But 2020 will see a paradigm shift in how teams and companies approach software development.
Company logo professionally milled and on its way to my new office
I will also, once there is more time, continue to sell and support software license components.
Quartex Media Desktop
The “Amibian.js” project is moving along nicely. The deadline is Q4 2019, but im hoping to wrap up the core functionality before that. So we are on track and kicking ass 🙂
More and more elaborate functionality is being implemented for the desktop
Here is an overview of work done this month:
- TSystemService application type has been created (node.js)
- TApplication now holds IPC functions (inter process communication)
- Running child processes + sending messages is now simplicity itself
- Database drivers are 90% done. Delete() and DeleteTable() functionality needs to be implemented in a uniform way
- Authentication is now a separate service
- Service database layer is finished (using SQLite3 driver by default)
- Authentication protocol has been designed
- Server protocol and JSON message envelopes are done
- Presently working on the client interface
- LDEF bytecode assembler has been improved
- Faster symbolic lookup
- Smarter register recognition
- Early support for stack-frames
- Fixed bug in parser (comma-list parse)
- QTX framework has seen a lot of work
- Large parts of the RTL sub-strata has been implemented
- UTF16 codec implemented
- QTX versions of common controls:
- TQTXButton
- TQTXLabel
- TQTXToolbar
- TQTXToolButton
- TQTXToolSeparator
- TQTXToolElement
- TQTXPanel
- TQTXCheckBox
- .. and much, much more
- Desktop changes
- Link Maker functionality has been added
- Handshake process between desktop and child app now runs on a separate timer, ensuring better conformity and a more robust initialization
- The Quartex Editor control has been optimized
- All redraw calls are now synchronized
- Canvas is created on demand, avoids flicker during initial redraw
- Support for DEL key + behavior
- Gutter is now rendered to an offscreen bitmap and blitted into the control’s canvas. The gutter is only fully rendered when cursor forces the view to change
I will continue to keep everyone up to date about the project. As you can understand, its a bit hectic right now so please be patient – it is turning into an EPIC environment!
Amiga Disrupt: talk from the heart
My previous article regarding the dreadful state the Amiga Kernel and OS finds itself in, primarily perpetuated by Italian company Cloanto, must have hit a nerve. My mailbox has been practically bombarded by people who are outraged by Cloanto (and Hyperion has got a fair bit of blame too). And indeed, there were errors made in that article (more about that below).
What I find strange, if not borderline insane, is how ingrained people are to their company or “team”. I have never understood people who watch soccer, who get physically upset over a game – or who demonstrate complete and utter loyalty to a team no matter how ridiculous that team might be. To me, soccer is just 22 grown men running around in their underwear chasing an inflated dead animal.
Thankfully, “Amiga hooligans” are a minority in the community. And it doesn’t really matter what topic you bring to the table, because they will oppose it either way. It’s what they do. The majority of the community are grown men and women with families, jobs and a life that has nothing to do with shared memories of the Commodore Amiga. And despite our differences we have one thing in common: a desire to see the system we grew up with flourish; a system that never failed and that despite its age has features and mechanisms that modern system lacks. It was management that failed, not the product.
As a developer, having to watch the brilliance of Amiga OS “rot on the wine” as the saying goes, is heartbreaking. The potential in the OS, even if we were to do a clean re-write, is astronomical. The ease of use alone for education, or as a low-cost alternative to Linux on embedded systems, has practical value far beyond gaming; which tragically is the only thing some people associate the technology with.
Points of view
The initial point of my article was not to paint Cloanto as the villain and Hyperion as the hero. I think everyone that has kept an eye on the Commodore saga and aftermath knows full well that none of the companies, both present and past, are without flaw. People don’t start companies for fun, but to do business. And the moment money is involved – human beings can demonstrate both excellence and selfishness. It’s human to make mistakes, and what ultimately matters is how we deal with them.
It all boils down to vantage-point. If your only ambition is to play some retro-games, then you will no doubt be happy with Cloanto’s Amiga Forever. If you enjoy software development and have coding as a hobby, then a full UAE setup, including cross compilers and real hardware will more than cover your needs.
So from those points of view, where you have already parked Amiga OS in the past as a dead system and hobby, I fully understand that you don’t care who did what, or the motives behind various strategic moves. Nothing wrong with that, people are different.
But what both those viewpoints have in common is that they are looking backwards to the past, rather than forward to a potential future. If you recognize that, and you yourself look to the future, then your expectations will be higher. You will care about how the IP is maintained, and also how the legacy is cared for. Legally it’s ultimately nobody’s business what Hyperion or Cloanto does with their intellectual property, but they have to remember that they are responsible for a computer legacy stretching back to the very beginning of home computers.
David’s book about what went on inside Commodore is quite a wake-up call. Go buy it ASAP!
The reason people refuse to throw Amiga out after so many years, is because the product was cut down before it’s time. Some compare it to the Betamax tragedy, where VHS despite being a lesser product ended up as the standard. And just like with the Amiga, it was not the product that was the determining factor in the tragedy, it was the lesser qualities of human beings. VHS allowed porn to be shipped en-mass on their format, while Betamax stuck to their principles and family values.
Commodore was thankfully not involved in anything as base, but if you take the time to read David Pleasance’s book: Commodore the inside story; you will discover that there were some monumental mistakes made in the name of, shall we say, “the lesser instincts of man“?. If you havent read his book then please do, then spend a few hours finding your jaw on the floor. It is absolutely shocking what went on behind closed doors in the company.
Mistakes in my post
The source of the mistake I wrote about, namely that of Acer’s ownership, is rooted in a simple misunderstanding. My focus was initially not on the ownership of the Amiga alone, but rather where has the Commodore patent portfolio gone? Commodore had been in business since 1954, and entered the computer market in 1979 with a MOS 6504 powered chess machine. A company with the level of growth and production over so many decades must have racked up some valuable patents, be they mechanical or electronic. I have never met Jack Trammell in person, but with regards to what I have read about the man, he would not miss an opportunity to make money or be whimsical about patents. So where did it all go?
Prior to my talk with Trevor Dickinson, I looked around to see who ended up with said portfolio (the proverbial needle in a haystack), I talked to several individuals in the community about this, googled, read articles – and was left with 3 potential candidates: HP, Acer and Asus.
While searching I came across the following video, and the ingress underlines Acer as the patent owner:
Acer is again mentioned as owning patents
When I then had a quick chat with Trevor and the name Acer turned up a third time, I saw no reason to question this. It was ultimately not the point of my post anyway.
The next question was to determine the relationship between said owner and those running the Amiga side of things (Cloanto and Hyperion). There were two logical possibilities: either these companies owned, in the true sense of the word, different parts of the legacy — or they functioned under a branding franchise. Meaning that they have been granted the right to evolve, sell and/or represent the Amiga name and technology with obligations of royalties. This is a pretty common business model, IBM being the archetypical example, so it would not be uncommon.
And that is ultimately the mistake. In retrospect I should have known there was no large company involved, because a stable corporation would never have allowed their IP to be mangled and dragged through the gutter like the Amiga have endured.
Having said that, it doesn’t really change much. I got an email saying that Cloanto have indeed given the authors of UAE money, which I hope is true because without the developers of UAE, the Amiga community would be abysmal. They have done 90% of the lifting, yet receive little praise for their work. But again – I was unable to find anything online where this could be confirmed.
It has also been stated that Amiga Inc was both tricked, abused and bullied by Hyperion. Yet the escapades of Amiga-Inc seem to have vanished into thin air:
“later that year, Amiga Inc. used some sleight of hand to escape a pending bankruptcy. Amiga sold its assets to a shell company called KMOS—a Delaware firm headquartered in New York—then renamed KMOS back to Amiga Inc. It tried to use these shenanigans to get out of the clause in its contract with Hyperion that would revert ownership of OS 4 if Amiga Inc. ever went under. Then, to top it off, Amiga sued Hyperion for not delivering OS 4 on time and demanded the return of all source code.” –Source: Ars Technica
Oh and then there was the “death threat” email. Where my post was said to be so diabolically crafted, so insiduius and evil – that i was responsible for possible death threats. I don’t even know how to respond to that, because the poo-nami that Cloanto is experiencing is the result of 15 years of silence; where the only communication has been to threaten Amiga users who accidentally shared a 512kb rom-file from the late bronze age with legal action. I think you gravely over-estimate my influence in the matter.
Right now Cloanto seem to run around pretending to be Santa. With promises of open-source and a future for their Amiga OS 4.1 (yes you read right) and that 3.1.4 is also theirs. First of all, Hyperion got that source-code as a part of the settlement with Amiga Inc (the quote above from ARS-Technica demonstrates how Amiga Inc treated Hyperion, not the other way around).
From a video posted by the 10 minute amiga retro-cast
Secondly, the Amiga OS 3.x source code has been available on the pirate bay for 4 years now? So if Cloanto indeed are so secure in their role as rightful heir to the Amiga throne, they can open source the code in a matter of hours. Just download, slap a GPL license on the files and push it out.
To nullify a 15-year-old settlement bound by contract, which is what must happen for them to have rights to their claims — that is something I wont hold my breath waiting for.
A viable business model
After my initial post people have dragged poor Trevor Dickinson into the debate, complaining to him about statements made by me. That is unfortunate because Trevor is not involved in our opinions at all. He even corrected me about mistakes I made in the previous article – and have absolutely not been a catalyst (quite the opposite!).
The Amiga history after the Commodore era is so convoluted, that his article series on the subject ended up spanning 12 issues of AF Magazine (!) Compare that to my two page brain fart. I also underlined that I had left out most of the details because rehashing the same tragedy over and over is paramount to explaining Game Of Thrones backwards in Sanskrit.
If we push all the details and who said what to the side for a moment, and look at the paths we have – it begins with a simple choice: you can look to the past and stick to “retro” computing. If that is the case then you will have no interest in anything I have to say, and that is fine. High five and enjoy.
If you look to the future, then suddenly we have some options before us: you have FPGA, like the FPGA-Arcade, the Vampire, MISTer and other, similar FPGA based systems. They have one thing in common and that is the 680×0 CPU.
Then you have software emulation, WinUAE being the trend-setter and various forks like UAE4Arm, FS-UAE and so on. This is perhaps the most versatile solution since it can do things difficult to achieve under real hardware.
Then we have the next generation and re-implementations. This is where Aros and it’s variations (AEROS, ARES et-al), Amiga OS 4.x and Morphos comes in.
I can’t see that we even need the legacy systems for much longer
And last but not least, cloud implementations like Amibian.js.
But in order for there to be any future where the core technology can grow, the technology has to serve a function in 2019. It doesn’t matter if the IPC layer is awesome, or that Amiga OS had REXX support 20 years before Mac OS. A modern system have to give users in this decade a benefit — otherwise there is no business model to talk about. And that is also my point. If we exclude web tech for now and look at the different paths, only two of them have the potential to deliver modern and unique functionality; and in my view that is Amiga OS 4 and Morphos.
FPGA will disrupt everything at some point
Vampire could perform a miracle and optimize their 68k architecture to the point where it can serve as a good embedded system, but even if possible, they are still held back by their dependency on classic Amiga OS. A partnership between Hyperion and Apollo would indeed be interesting, who knows. Although I would love to see the Apollo team fork Aros and shape that into what it could become with a bit of work.
Morphos is rumored to be moving their codebase to x86. This is just a rumour and I havent seen any documentation around that. If this is true then I feel it is a mistake, because NVidia and roughly 100 other major players are about to attack Intel on all fronts with RISC-V – and ARM is set to replace x86 in consumer electronics faster than most expected. Apple just announced that ARM based laptops are in the making.
I should add that this is also why I decided to write Amibian.js using web technology, because regardless of which CPU or architecture that becomes dominant in the next decade, web tech will always be there. So it allows us to abstract away the costly dependency on hardware, and instead focus on functionality.
PPC for the win?
In an interesting twist of fate, PPC could actually come out far better than anticipated – but not in the way you might think. Work is being done to make PPC a first class FPGA citizen. FPGA is fantastic in many ways, but it’s the intrinsic abillity to “become” whatever technology you describe that is revolutionary.
While it’s still in its infancy, the potential is there to render instruction-sets and architectures a preference rather than a requirement. If anything, the Vampire IV is a demonstration of just that.
So code currently bound to PPC could use FPGA as an intermediate solution while the codebase is ported to more viable platforms.
So whats the problem?
The next question then becomes: what exactly is stopping the owners from moving forward? Why dont the companies that hold the various IP’s roam silicon-valley in search of funding? And it’s here that we face the situation I briefly painted a picture of in my last post: they are in a perpetual stale-mate.
And in my view (as a developer looking forward) Cloanto, whose primary focus is to provide for the legacy market, is constantly getting in the way of Hyperion – which is looking at the future. As far as innovation and managing the legacy of Commodore is concerned, Cloanto has been asleep at the wheel for over a decade. They only woke up when it could cash-in on its C64 assets. I have no number as to how many c64 mini’s have been sold around the world, but its been a massive success. And it would be foolish to think that they have no plans to repeat the success with an Amiga model — effectively hammering the final nail in the coffin. After that, the Amiga is forever a legacy system.
Well. This case is already boring the hell out of me, so I will just leave them to it.
But looking at the various paths forward, from where I stand Hyperion and OS 4.x is the only viable business model. Providing the goal is to bring the technology back into the consumer-market and evolve the technology as an alternative to Windows, OS X and Linux. If the goal is just milk the system one final time, then I would say they are already there.
I honestly could not care less at this point. They have been asleep for so long, that they have become irrelevant. The future is in cloud, clustering and hardware abstraction — and Amibian.js is already far more interesting than anything cloanto has on offer.
But make no mistake: If the parties involved dont get their shit together, come 2022 and we will implement a native OS ourselves and open source it through torrents. The Quartex consortium is deadly serious about this. The new QTX is made up of members from various established groups back in the day, now in our 40s and 50s. Like all amiga users we have tolerated this for two decades, but enough is enough. Unlike the average gamer most of us are professional developers with decades of experience.
They have until 2022, if nothing has changed, we will finish this for them
And that was my five cents on that matter, and the last post I will do on this dumpsterfire of a topic.
Repository updates
As most know by now, I was running a successful campaign on Patreon until recently. I know that some are happy with Patreon, but hopefully my experience will be a wakeup call about the total lack of rights you as a creator have – should Patreon decide they don’t understand what you are doing (which I can only presume was the case, because I was never given a reason at all). You can read more about my experience with Patreon by clicking here.
Setting up repositories
Having to manually build a package for each tier that I have backers for would be a disaster. It was time-consuming and repetitive enough to create packages on Patreon, and I don’t have time to reverse engineer Patreon either. Which I might do in the future and release as open-source just to give them a kick in the groin back.
To make it easier for my backers to get the code they want, I have isolated each project and sub-project in separate repositories on BitBucket. This covers Delphi, Smart Pascal, LDEF and everything else.
The CloudRipper architecture is coming along nicely. Here running on ODroid XU4
I’m just going to continue with the Tiers I originally made on Patreon, and use my blog as the news-center for everything. Since I tend to blog about things from a personal point of view, be it for Delphi, JavaScript or Smart Pascal — I doubt people will notice the difference.
So far the following repositories have been setup:
- Amibian.js Server (Quartex Web OS)
- Amibian.js Client
- HexLicense
- TextCraft (source-code parser for Delphi and Smart Pascal)
- UAE.js (a fork of SAE, the JS implementation of UAE)
I need to clean up the server repository a bit, because right now it contains both the server-code and various sub projects. The LDEF assembler program for example, is also under that repository — and it belongs in its own repository as a unique sub-project.
The following repositories will be setup shortly:
- Tweening library for Delphi and Smart Pascal
- PixelRage graphics library
- ByteRage bugger library
- LDEF (containing both Delphi and Smart Pascal code)
- LDEF Assembler
It’s been extremely busy days lately so I need to do some thinking about how we can best organize things. But rest assured that everyone that backs the project, or a particular tier, will get access to what they support.
Support and backing
I have been looking at various ways to do this, but since most backers have just said they want Paypal, I decided to go for that. So donations can be done directly via paypal. One of the new features in Paypal is repeated payments, so setting up a backer-plan should be easy enough. I am notified whenever someone gives a donation, so it’s pretty easy to follow-up on.
Updates used to be monthly, but with the changes they will be ad-hoc, meaning that I will commit directly. I do have local backups and a local git server, so for parts of the project the commits will be issued at the end of each month.
While all support is awesome, here are the tiers I used on Patreon:
- $5 – “high-five”, im not a coder but I support the cause
- $10 – Tweening animation library
- $25 – License management and serial minting components
- $35 – Rage libraries: 2 libraries for fast graphics and memory management
- $45 – LDef assembler, virtual machine and debugger
- $50 – Amibian.js (pre compiled) and Ragnarok client / server library
- $100 – Amibian.js binaries, source and setup
- $100+ All the above and pre-made disk images for ODroid XU4 and x86 on completion of the Amibian.js project (12 month timeline).
So to back the project like before, all you do is:
- Register with Bitbucket (free user account)
- Setup donation and inform me of your Bitbucket user-name
- I add you on BitBucket so you are granted access rights
Easy. Fast and reliable.
The QTX RTL
Those that have been following the Amibian.js project might have noticed that a fair bit of QTX units have appeared in the code? QTX is a run-time library compatible with Smart Mobile Studio and DWScript. Eventually the code that makes up Amibian.js will become a whole new RTL. This RTL has nothing to do with Smart Mobile Studio and ships with its own license.
QTX approaches the DOM in more efficient way. Its faster, smaller and more powerful
Backers at $45 or beyond access to this code automatically. If you use Smart Mobile Studio then this is a must. It introduces a ton of classes that doesn’t exist in Smart Pascal, and also introduces a much faster and clean visual component framework.
If you want to develop visual applications using QTX and DWScript, then that is OK, providing the license is respected (LGPL, non commercial use).
Well, stay tuned for more info and news!
Quartex: Mali GPU glitches
EDIT: I did further testing after this article was written, and believe the source of this to be about heat. Even with extra fans, running games like Tyrian (asm.js) that are extremely demanding, plus resizing a graphics intensive windows constantly, the temperature reached 71 degrees C very quickly. And this was with two cabinet fans helping the built-in fan to cool the device. It is thus not unthinkable that when running solo (no extra fans) that the kernel shut the device down to not cook the chipset. Which also explains why the device wont boot properly afterwards (the device is still hot).
Glitches
Something really strange is happening on Chrome and Firefox for ARM. JavaScript is not supposed to be able to take down a system, and in this case it’s neither an attempt as such either — yet for some reason I have managed to take down the ODroid XU4 with both Chrome and Firefox lately.
ODroid XU4
I guess I should lead with that I’m not able to replicate this on x86. One of the things I really love about the ODroid XU4 is that it’s affordable, powerful and probably the only SBC I have used that runs stable on the mali GPU. As you probably know I tested at least 10 different SBC’s back in 2018, and whenever there was a mali GPU involved, the product was either haunted by instabilities or lacked drivers all together.
Since the codebase for Chrome (and I presume Firefox) is ultimately the same between platforms, it leaves a question-mark about the ODroid. It is by far the most stable SBC I have tested so far (except for the PI, which is sadly underpowered for this task), but stable doesn’t mean flawless. And to be honest, Amibian.js is pushing web tech to the very limits.
Not Mali again
The reason I suspect the mali to be the culprit behind all this, is because the “bug” if we can call it that, happens exclusively during resize. So if there is a lot going on inside a desktop-window, you can sometimes provoke the ODroid to cold-crash and reboot. You actually have to power the board down and switch it back on for it to boot properly.
Cloudripper ~ 5x ODroid XU4 [40 cores] in a PICO 5h cube
The challenge is that there is no way to debug or catch this error, because when it occurs the whole system literally goes down. There is no exception thrown, nor is the browser process terminated (not even a log entry, so it’s a clean-cut) — the system reboots on the spot. Since it fails on reboot when opening X (setting a screen-mode) I again point the finger at the GPU. Somehow a flag or lock survives the cold-reboot and that’s why you have to manually switch it off and on again.
This is the exact problem that made the NanoPI Fire useless. It only shipped with Android embedded drivers. The X drivers could hardly open a display without crashing. Such a waste of a good cpu.
x86 as head
ODroid is perfect for a low-cost Amibian.js experience, but I was unsure if it would handle the payload. Interestingly it handles it just fine and even with a high-speed action game running + background tasks we are not using 50% of the CPU even.
Ram is holding up too, with memory consumption while running Tyrian + having a few graphics viewers open, is at a reasonable 700 mb (of 2 gigabyte in total).
Tyrian jogs along at 45 fps ~ that is not bad for a $45 SBC
Right now this strange error is rare, but if it continues or grows into a problem (chrome is hardly useable at all, only firefox) then I have no option than to replace the master sbc in the cluster with something else. The x86 UP board is more than capable, but it would be a shame to break the price range because of that (excuse my language) crap mali GPU. I honestly don’t understand why board makers insist on using a mali. Every board that has a mali is haunted by problems and get poor reviews.
It will be exciting to check out the dragonboard, although I fear 1Gb memory will not be enough for smooth operation. Not without a sata interface and a good swap-file.
Android and Delphi
One alternative is to switch to Android and use Delphi to code a custom Chromium Embedded webview. I am hoping to avoid the overhead of Android, but Delphi would definitively be a bonus with Android embedded (“Android of things”).
We will see.
Leaving Patreon: Developers be warned
As a person I’m quite optimistic. I like to think the glass is half-full rather than half-empty. I have spent over a decade building up a thriving Delphi and C++ builder community on social media, I have built up a rich creative community for node and JavaScript on the side — not to mention retro computing, embedded tech and IOT. For better or for worse I think most developers in the Embarcadero camp have heard my name or engage in one of the 12 groups I manage around the world on a daily basis. It’s been hard work but man, it’s been worth every minute. We have so much fun and I get to meet awesome coders on a daily basis. It’s become an intrinsic part of my life.
I have been extremely fortunate in that despite my disadvantage, a spine injury in 2012 – not to mention being situated in Norway rather than the united states; despite these obstacles to overcome I work for a great American company, and I get to socialize and have friends all over the planet.
The global village is the concept, or philosophy, that technology makes it possible no-matter where you live, to connect and be a part of something bigger. You don’t have to be a startup in the san-francisco area to work with the latest tech. Sure a commute from Burlingame to Redwood beats a 14 hour flight from Norway any day of the week — but that’s the whole idea: we have Skype now, and Slack and Github; you don’t have to physically be on location to be a part of a great company. The only requirement is that you make yourself relevant to your field of expertise.
Patreon, a digital talent agency
Patreon is a service that grew straight out of the global village. If the world is just one place, one great big family of human beings with great ideas, then where is the digital stage that helps nurturing these individuals? I mean, you can have a genius kid living in poverty in Timbuktu that could crack a mathematical problem on the other side of the globe. The next musical prodigy could be living in a loft in Germany, but his or her voice will never be heard unless it’s recognized and given positive feedback.
“The irony is that Patreon doesn’t even pass their own safety tests. That should make you think twice about their operation”
My examples are extremes I agree, most people on Patreon are like me, creative but absolutely not cracking math problems for Nasa; nor am I singing a duet with Bono any time soon. But that’s the fun thing about the world – namely that all things have value when put in the correct context. Life is about combinations, and you just have to find one that works for you.
The global village, the idea of unity through diversity
The global village is this wonderful idea that we can use technology to transcend the limitations the world oppose on us, be they nationality, color, gender or location. Good solutions know no bounds and manifests wherever a mind welcomes it. Perhaps a somewhat romantic idea, if not naive, but it seems the only reasonable solution given the rapid changes we face as a species.
In my case, I love to make software components in my spare time. My day job is packed and I couldn’t squeeze in more work during the weekdays if I wanted to, so I only have a couple of hours after-work and the weekends to “do my thing”. So being a total geek I relax by making components. Some play chess, the guitar or whatever — I relax by coding something useful.
Obviously “code components” are completely useless to anyone who is not a software developer. The relevance is further clipped by the programming-language they are written for, and ultimately the functionality they provide. Patreon for me was a way to finance the evolution of these components. A way of self motivating myself to keep them up to date and available.
I also put a larger project on Patreon, namely the cloud desktop system people know as “Amibian.js” or “Quartex Web OS”. Amibian being the nickname, or codename.
Patreon seemed like the perfect match. I could take these seemingly unrelated topics, Delphi and C++ builder specific components and a cloud architecture, and assign each component and project to separate “tiers” that the audience could pick from. This was great! People could now subscribe to the tier’s they wanted, and would be notified whenever there was an update or new features. And I could respond to service messages in one place.
The Tier System
The thing about software is that it’s not maintained on infinite repeat. You don’t fix a component that is working. And you don’t issue updates unless you have fixed bugs or added new functionality. A software subscription secures a customer access to all and any updates, with a guarantee of X number of updates a year. And equally important, that they can get help if they are stuck.
“when you are shut down without so much as an explanation, with nothing but positive feedback, zero refunds and over 1682 people actively following the progress — that is utterly unacceptable behavior”
I set a relatively low number of guaranteed updates per year for the components (4). The things that would see the most updates were the Rage Libraries (PixelRage and ByteRage) and Amibian.js, but not until Q3 when all the modules would come together as a greater whole — something my backers are aware of and have never had a problem with.
Amibian.js running on ODroid XU4, a $45 single board computer
The tiers I ended up with was:
- $5 – “high-five”, im not a coder but I support the cause
- $10 – Tweening animation library
- $25 – License management and serial minting components
- $35 – Rage libraries: 2 libraries for fast graphics and memory management
- $45 – LDef assembler, virtual machine and debugger
- $50 – Amibian.js (pre compiled) and Ragnarok client / server library
- $100 – Amibian.js binaries, source and setup
- $100+ All the above and pre-made disk images for ODroid XU4 and x86 on completion of the Amibian.js project (12 month timeline).
Note: Each tier covers everything before them. So if you pick the $35 tier, that also includes access to the license management system and the animation library.
As you can see, the tier-system that is intrinsic to Patreon, solves the software subscription model elegantly. After all, it would be unreasonable to demand $100 a month for a small component like the Tweening library. A programmer that just needs that library and nothing else shouldnt have to pay for anything else.
Here is a visual representation, showing graphically why my tiers are organized as they are, and how they all fit into a greater whole:
The server-side aspect of the architecture would take days to document, but a general overview of the micro-service architecture is fairly easy to understand:
Each of the tiers were picked because they represent key aspects of what we need to create a visually pleasing, fast and reliable, distributed (each part running on separate machines or boards) cloud eco-system. Supporters can just get the parts they need, or support the bigger project. Everyone get’s what they want – all is well.
The thing some people don’t grasp, is that you are not getting something to just put on Amazon or Azure, you are getting your own Amazon or Azure – with source code! You are not getting services, you are getting the actual code that allows YOU to set up your own services. Anyone with a server can become a service provider and offer both hosting and software access. And they can expand on this without having to ask permission or pay through the nose.
So it’s a little bit bigger than first meets the eye.
I Move In Mysterious Ways ..
Roughly 3 weeks ago I was busy preparing the monthly updates.
Since each tier is separate but also covers everything before it (like explained above) I have to prepare a set of inclusive updates. The good news is that I only have to do this once and then add it as an attachment to my posts. Once added I can check of all the backers in that tier. I don’t have to manually email each backer, physically copy my songs or creations onto CD and send it – we live in the digital age as members of the global village. Or so i thought.
So I published two of the minor cases first: the full HTML5 assembly program, that can be run both inside Amibian.js as a hosted application — or as a solo program directly in the browser. So here people can write machine-code in the browser, assemble it to bytecodes, run the code, inspect registers, disassemble the bytecodes and all the normal stuff you expect from an assembler.
This update was special because the program contained the IPC (inter process communication) layer that developers use to make their programs talk to the desktop. So for developers looking to make their own web programs access the filesystem, open dialogs (normal system features), that code was quite important to get!
Although published, none of my backers could see them due to the suspended status
The second post was a free addition, the QTX library which is an open-source RTL (run time library) compatible with the Smart Pascal Compiler. While not critical at this juncture, several of my backers use Smart Mobile Studio, and for them to get access to a whole new RTL that can be used for open-source, is very valuable indeed.
I was just about to compress the Amibian.js source-code and binaries when I got a message on Facebook by a backer:
“Dude, your Patreon is shut down, what is happening?”
What? hang on let me check i replied, and rushed into Patreon where the following header greeted me:
What the hell Patreon? I figured there must be some misunderstanding and that perhaps I missed an email or something that needed attention. I get close to 50 emails a day (literally) so it does happen that I miss one. I also check my spam folder regularly in case my google filters have been careless and flagged a serious email as spam. But there was nothing. Not a word.
Ok, so let’s check the page feedback, has there been any complaints? Perhaps a backer has misunderstood something and I need to clear that up? But nope. I had nothing but positive feedback and not even a single refund request. In fact the Amibian.js group on Facebook has grown to 1,662 members. Which shows that the project itself holds considerable interest outside software development circles.
Well, let’s get on this quickly I thought, so I rushed off an email asking why Patreon would do such a thing? My entire Patreon page was visibly marked with the above banner, so my backers never even saw the updates I had issued.
Instead, the impression people would get, was that I was involved in something so devious that it demanded my account to be suspended. Talk about shooting first and asking later. I have never in my life seen such behavior from a company anywhere, especially not in the united states; Americans don’t take kindly to companies behaving like bullies.
Just Contact Support, If You Can Find Them
To make a long story short it took over a week before Patreon replied to my emails. I sent a total of 3 emails asking what on earth would have prompted them to shut down a successful campaign. And how they found it necessary to slander the project without even informing me of the problem. Surely a phone call could have sorted this up in minutes? Where I come from you pick up the phone or get in contact with people before you flag them in public.
Sounds great, sadly it’s pure fiction
The response I got was that “some mysterious activity had been reported on my page”, and that they wanted my name, address, phone number and credit card (4 last digits). Which I found funny because with the exception of credit-card details, I always put my name, address, phone numbers and email etc. at the head of my letters.
I’m not a 16-year-old kid working out of a garage, im a 46-year-old established software developer that have worked as a professional for close to 3 decades. Unlike the present generation I moved into my first apartment when I was 16, and was working as an author for various tech magazines by the time I was 17. I also finished college at the same time and went on to higher-education (2 years electrical engineering, 3 years arts and media, six years at the university in oslo, followed by 4 years of computer science and then certifications). The focus being, that Patreon is used to dealing with young creators that will go along with things that grown men would not accept.
But what really piss me off, was that they never even bothered to explain what this “mysterious behavior” actually was? I write about code, clustering, Delphi, JavaScript and bytecodes for christ sake. I might have published updates and code wearing a hoodie at one point, in a darken room, listening to Enigma.. but honestly: there is not enough mystery in my life to cover an episode of Scooby-Doo.
Either way, I provided the information they wanted and expected the problem to be resolved asap. Two days at themost. Maybe three, but that was pushing it.
It’s now close to 3 weeks since this ridiculous temporary suspension occurred, and neither have I been given any explanation to what I have done, nor have they removed the ban on the content. I must have read their guidelines 100 times by now, but given the nature of their ruling (which are more than reasonable), I can’t see that I have violated a single one:
- No pornography and adult content
- No hate speech against minorities or forms of religious extremism
- No piracy or spreading copyrighted material
- No stealing from backers
Let’s go over them one by one shall we?
Pornography and adult content
Seriously? I don’t have time to loaf around glaring at naked women (i’m a geek, I look weird enough as it is), and after 46 years on this planet I know what a woman looks like nude from every possible angle; I don’t need to run around like a retard posting pictures of body parts. And if you are talking about me — good lord is there a marked for hobbits? Surely the world has enough on it’s plate. Sorry, never been huge on porn.
And for the record, porn is for teenagers and singles. The moment you love someone deeply, the moment you have children together — it changes you profoundly. You get a bond to your wife or girlfriend that makes you not want to be with others. Not all men are into smut, some of us are invested more deeply in a relationship.
Hate speech and religious extremism
Hm, that’s a tough one (sigh). Did you know that one of my best friends is so gay – that he began to speculated that he actually was a liquid? He makes me laugh so bad and he’s probably the best human being I have ever met. I actually went with him on Pride last year, not because i’m gay but because he needed someone to hold the other side of the banner. That’s what friends do. Besides, I looked awesome, what can I say.
As for religion I am a registered Tibetan Buddhist. I believe in fluffy pillows, comfy robes, mother nature and quite frankly I find the world inside us far more interesting than the mess outside. You cant be extreme in Buddhism: “Be kind now, or ill hug you until you weep the tears of compassion!”. Buddhism sucks as an extreme doctrine.
So I’m going to go out on a limb and say nuuuu to both.
Piracy and copyrighted material
Eh, I’m kinda writing the software from scratch before your eyes (including the run-time-library for the compiler), so as far as worthy challenges go, piracy would be the opposite. I am a huge fan of classical operating-systems though, like the Amiga; But unlike most people I actually took the time to ask permission to use a OS4 inspired CSS theme-file.
The Amibian.js project is well organized and I have worked systematically through a well planned architecture. This is not some slap-dash project made for a quick buck
Most people just create a theme-file and don’t bother to ask. I did, and Trevor Dickinson was totally cool about it. And not a single byte has been taken or stolen from anyone. The default theme file is inspired by Amiga OS 4.1, but the thing is: the icons are all freeware. Mason, the guy that did the OS icons, have released large sets of icons into GPL. There is also a website called OS4Depot where people publish icons and backdrops that are free for all.
So if this “mysterious activity” is me posting a picture of a picture (not a typo) of an obscure yet loved operating-system, rest assured that it’s not violating anyone.
Stealing from backers
That they even include this as a point is just monumental. Patreon is a service established to make that impossible (sigh); meaning that the time-frame where you deliver updates or whatever – and the time when the payout is delivered, that is the window where backers can file a complaint or demand a refund.
And yes, complaints on fraud would indeed (and should!) flag the account as potentially dubious — but again, I have not a single complaint. Not even a refund request, which I believe is pretty uncommon.
And even if this was the case, shutting down an account without so much as a dialog in 2019? Who the hell becomes a thief for 600 dollars? Im not some kid in a garage, I make twice that a day as a consultant in Oslo, why the heck would I setup a public account in the US, only to run off with 600 bucks! I have standing offers for projects continuously, I havent applied for a job since the 90s – so if I needed some extra money I would have taken a side project.
I even posted to let my backers know I had a cold last month just to make sure everyone knew in case I was unavailable for a couple of days. Truly the tell-tell sign of a criminal mastermind if I ever saw one ..
Sorry Patreon, but your behavior is unacceptable
Hopefully your experience with Patreon has not been like mine. They spent somewhere in the range of 5 weeks just to register me, while friends of mine in the US was up and running in less than 2 days.
We are now 3 weeks into a temporary suspension, which means that most of my backers will run out of patience and just leave. It sends a signal of being whimsical about other people’s trust, and that people take a risk if they back my project.
At this point it doesn’t matter that none of these thoughts are true, because they are thoughts that anyone would think when a project remains flagged for so long.
What should scare you as a creator with Patreon though, is that they can do this to anyone. There is nothing you can do, neither to prove your innocence or sort out a misunderstanding — because you are not even told what you allegedly have done wrong. I also find it alarming that Patreon actually doesn’t have a phone-number listed, nor do they have offices you can call or reach out to.
The irony is that Patreon doesn’t even pass their own safety tests. That should make you think twice about their operation. I had heard the rumors about them, but I honestly did not believe a company could operate like this in our day and age. Especially not in the united states. It undermines the whole spirit of US as a technological hub. No wonder people are setting up shop in China instead, if this is how they are treated in the valley.
After this long, and the damage they have caused, I have no option than to inform my backers to terminate their pledges. I will have to relocate my project to a host that has more experience with software development, and who treats human beings with common decency and respect.
If I by accident had violated any of their guidelines, although I cannot see how I could have, I have no problem taking responsibility. But when you are shut down without so much as an explanation, with nothing but positive feedback, zero refunds and over 1682 people actively following the progress — that is utterly unacceptable.
It is a great shame. Patreon symbolized, for a short time, that the global village had matured into more than an idea. But I categorically refuse to be treated like this and find their modus-operandi insulting.
Stay Well Clear
If you as a developer have a chance to set up shop elsewhere, then I urge you to do so. And make sure your host have common infrastructure such as a phone number. Patreon have taken the art of avoiding direct contact to a whole new level. It is absolutely mind-boggling.
I honestly don’t think Patreon understands software development at all. Many have voiced more sinister motives for my shutdown, since the project obviously is a threat to various companies. But I don’t believe in conspiracies. Although, if Patreon does this to enough creators on interval, the interest rates from holding the assets would be substantial.
It could be that the popularity of the project grew so fast that it was picked up as a statistical anomaly, but surely that should be a good thing? Not to mention a potential case study Patreon could have used as a success story? I mean, Amibian.js didn’t get up and running until october, so stopping a project 5 months into a 12 month timeline makes absolutely no sense. Unless someone did this on purpose.
Either way, this has been a terrible experience and I truly hope Patreon get’s their act together. They could have resolved this with a phone-call, yet chose to let it fester for almost a month.
Their loss.
Hyperion vs Cloanto, the longest running lawsuit in the history of computing?
Delphi and C++ builder developers will probably not have much interest in this, but as far as general IT news goes, this one is attracting interest far and wide due to the sheer absurdity involved. To be honest I also think that the case itself serves as a warning to companies and developers in general, because this truly is the best example of how bad things can go if you don’t manage your patents and rights properly.
So while I’m loving Delphi’s 24th birthday festivities, I find the ongoing lawsuits so amazing that I have to write a few words.
[Edit]: To make the case even remotely understandable for people that have never read about it before, I have left out a ton of details. The whole Amiga Inc scandal (which I believe ordered production of OS4 to begin with?), Eyetech, H&P, the loss of the Amiga OS 3.9 source code. The gist of the post here is not to dig into the details (also known as “the rabbit hole” in the community), but to give a short recount of the highlights leading up to the present situation – and to underline that people who still care for the system, the Amiga community, is beyond fed-up with this. I hope all parties get their act together and find a way to co-exist. For those that want to dig into the gory details spanning three decades, there is always the Amiga documentation project.
Some context
Long story short, back in the early 90s Commodore, a company that for close to two decades ranked as a giant of computing, collapsed. Years of mismanagement, poor leadership, if not outright shameful, had taken its toll on the once fierce giant; And as the saying goes: the bigger they are, the harder they fall. And boy did Commodore fall.
Commodore ranked side-by side with the biggest names in the industry
What people often forget is that tech-companies have two types of currencies. The first is what consumers consider valuable; things like the products they make, how much money is in the bank, the state of their inventory, good partners and retailers — all points of importance when running a business.
Major players though couldn’t care less about these factors, not unless they align with their own needs. So from a PC company’s perspective, getting rid of the Amiga and butchering Commodore for patents was a spectacular win. Because, and here we get into the nasty parts: for an already established competitor, a dead tech company has one asset and one asset only: namely their patent-portfolio.
So all that buying and selling we saw in the 90s, with Amiga changing hands left and right, had nothing to do with saving the Amiga. The Commodore legacy was reduced to a piece of meat and thrownto the wolves, each ripping into its patents left and right. So while graphic, the piece of meat in this analogy held an estimated value of a billion dollars.
Patents are valuable because they represent repeated income and a level of financial security unline ordinary currency. Large companies use patent portfolios in combination with their insurance. IBM is more or less the archetypical example of this. They remain one of the richest companies in the world, but spend their time tinkering with super-computers and science experiments. “Big Blue” haven’t “worked” in the true sense of the word since they started licensing out PC as a platform. They own the patents for pretty much everything we know as a PC today, and don’t need to compete. They make a fortune just sitting there.
Climbing up the rabbit hole
Gateway and Escom both tried to save themselves using the Amiga patents, but they failed
When Commodore fell, the vultures moved in quickly. People have focused so much on the Amiga computer and branding aspect of Commodore, that we often neglect that the true value of such a giant was never the end-product, but the intrinsic values of their patents and technological inventions.
Very few knew the identity of the party now in possession of the Commodore patent portfolio until quite recently. It caused quite a stir online when I published the name of the owner last year (both on this blog and Amiga Disrupt on Facebook).
Just to underline: this information have never been secret or anything of the sorts. It’s just a type of information ordinary people wouldn’t know where to find (myself included). You have to know where to look and what to look for. And while I have some experience with copyright cases and intellectual property – I would never have found it without a heart to heart with Trevor Dickinson. The major shareholder in Aeon, which produces the Next Generation Amiga system (x5000 and the upcoming A1222). He kindly helped me through the avalanche of older court documents and pointed me to an article series in AF Magazine that I had no idea even existed.
I should also stress that I have no special friendship with Trevor. I have talked to him on various occasions and we share a passion for the Amiga system. He has always been very kind, but I don’t know him personally. Nothing I write here is done in his favour or out of some form of loyalty. I simply find that A-EON and Hyperion’s plans and products makes the most sense in 2019.
When the mysterious owner of Commodore and Amiga turned out to be Acer my jaw dropped. They had been sitting on the patents for all these years without making a sound. From Acer’s point of view the Amiga computer is worthless and they wouldn’t give a cup of coffee for the Amiga name or its legacy. So the Amiga name and legacy code was sold off long ago. Acer handles technological patents that commodore deviced, from PET to 3A no doubt. Amiga as a platform is uninteresting to them.
How Acer got a hold of the portfolio can only be speculated on, but I would imagine they snapped them up when Escom went under. How much of the original portfolio remains intact is anyone’s guess. The classical Amiga OS source-code was, as we know, acquired by Hyperion from Amiga Inc years ago. That was the 3.1 version. Interestingly the 3.9 version was bought by H&P (a german company) and was sadly lost when they existed the Amiga market permanently.
Workbench and hipsters
For those that haven’t read or followed up on the “Commodore case”, the license holders mentioned above (A-EON, Hyperion, Cloanto), have been at each other’s throats since the brits annexed India. Which is why this case has become interesting for others as well.
Nobody under 33 years of age would associate this with Commodore or Amiga.
To give you some examples of the epic battles at hand: they have argued in court over the right to use a checkered bathing ball, you know those you can buy almost anywhere and that resemble a french table-cloth? Oh yes I kid ye not.
They have gone to court over the misuse of said bathing apparatus, the misrepresentation of the ball, who owns the ball, it’s buoyancy – and let us not forget trademarking the word “Workbench” (the name of the desktop system the Amiga uses). A word today only used by hipsters in meth-labs and tool-time-tim wannabe’s on YouTube. The absurdities are so dense you could bottle them.
If we look at the many struggles since Commodore went under from a bird’s eye perspective, we are essentially seeing the same lawsuit on infinite repeat (with a few variations here and there). I got married, I had kids and 15 years later I got divorced. And when I got back they were still at it! Good god guys, what a complete and utter waste of time, resources and talent (The lawsuits not my marriage. Well maybe both), not to mention counter productive! If anything these frequent lawsuits are destroying what both parties are trying to protect. Although I question if one of them indeed are.
If I was to go back to school and re-invent myself, I would become an author. All I had to do to was take the Commodore story and place it in middle-earth, give the people involved pointy ears, brutal weaponry and silly names and voila! A tale that would make Tolkien himself weep; because great as his imagination was, never could he have concocted such a story. Not even Keith Richards if we let him loose in a pharmacy on “take all the drugs you can carry day” – could make up a timeline as insane as the Commodore aftermath.
Lawsuits 1-0-1: Que bono?
To catch you up with the present events, let’s just go through the basics first.
It can be difficult to distinguish between Hyperion and Aeon, so lets start with a few words about that. Hyperion is ultimately a software company. They started (if I recall correctly) as software house porting PC games to the Amiga platform.
I previously wrote that Trevor was the major shareholder in both companies, that was actually wrong, he holds a very small role in Hyperion. But who owns what here is ultimately pointless. The relationship between Hyperion and A-EON is that Hyperion represents the software branch, and A-EON is the hardware branch. And combined they make out the owners and producers of what is commonly called “Next Generation” Amiga machines.
A-EON and Hyperion hold the rights to develop Amiga OS, covering both the classical 68k version and the NG models which are PPC based. Cloanto have only sales rights, which are limited to the legacy 68k ROM kernel files, and workbench. That is ultimately what separates these two groups. So even though there are 3 companies involved, it’s easier to regard them as two separate entities.
And yes we could argue that OS4 was instigated by Amiga Inc earlier, but i’m trying to keep this readable for people that haven’t read anything about this silliness before, so i’m skipping all of that.
Amiga OS is loved by many, but to be frank it’s reached the point that fighting over it has long since passed. A teenager today knows PSX, XBox and completely different brands
Until recently Aeon and Hyperion have focused completely on their Next Generation system. Aeon creates the hardware and Hyperion does the software. Hyperion also offers the older legacy roms and Workbench in their webshop. But until recently they have been more interested in selling next-generation software and machines.
Cloanto have been exclusively about legacy. They have no license that involves software development, and are for all means an purposes a retro retailer (or undertaker if you will). They sell old Commodore stuff, and that’s it. So while they have argued like cats and dogs over absolutely everything, like that worthless boing ball and the name “workbench”, they at least managed to co-exist somehow.
That was, until Hyperion listened to the Amiga Community and released an update for the 68k platform. Which is perfectly within their rights to do. They have a license that covers both 68k and PPC. Acer has set a clause (from what I can tell) that they are not allowed to touch x86, but as far as 68k and PPC is concerned — Hyperion is well within their rights to issue an update. After all they own the source-code for Amiga OS 3.1 which I mentioned above, Cloanto does not.
The response from the community was quite frankly outstanding. Finally a proper update for both Workbench and the kernel! Everyone was ecstatic and the whole scene was filled with positive hopes that things were finally moving forward. This was after all the first real update since Napoleon was in office!
Cloanto however, not so much. Because even though they share the sales license with Aeon, they have no rights to the new software created. They don’t make a penny on the new 68k kernel (rom files) or the new Workbench. They can continue to sell the older variations of Amiga OS, but they have no legal right to software written and issued in 2018. Cloanto responded like they always have, by issuing a lawsuit.
So the reason Cloanto took Hyperion to court for the 13th thousand time, has nothing to with open-source (a rumour that was planted before Xmas). It is motivated purely by greed and the fear that the Amiga might actually spring back to life.
And this is where we get to the nasty parts
Legacy software undertakers
Legacy software is not unlike the undertaking business
First of all, and I want to make this crystal clear: Cloanto’s entire business model rests on the Amiga remaining dead. In a bizarre twist of irony, the self-proclaimed caretakers of Amiga actually face financial ruin if the Amiga ever became popular or rose from the grave. Stop and think about that for a moment: They make money on the Amiga remaining a dead system.
The only product Cloanto have actually produced, is a pixel paint program called PPaint, which was awesome back in the previous century.
The state of affairs for the past 18 years, is that Cloanto depends completely an emulator, UAE, short for “The Unix Amiga Emulator”, when it comes to the Amiga . Which ironically is not Cloanto’s work at all, but an emulator created by Bernd Schmidt, Toni Wilen and Mathias Ortmann; neither have received a penny despite Cloanto profiting on their work for close to two decades (!)
The selling of legacy Commodore software I have no problem with at all. But what bakes my noodle is forking UAE and selling it for profit without giving something back to its original authors? I have yet to see the source-code for Amiga Forever on Github for example? The laws of GPL are pretty straight forward. I’m not saying that the source code does not exist, i’m simply saying that Cloanto has gone out of their way to keep it hidden.
Sure it may be legal but I find it somewhat tasteless. profiting on UAE for all those years, and not even a symbolic sum for the guys that keep UAE going? I mean, had they actively participated and contributed to the UAE codebase I would have applauded them for it. Sadly Cloanto presents itself as a blatant opportunist more than a preserver. They say one thing, but their actions speak of something else entirely.
And don’t get me wrong, Hyperion and Aeon have more than enough mistakes on file. But when comparing Hyperion’s mistakes against Cloanto, remembering that these two have an obligation to represent the Amiga legacy to the best of their ability — you cannot help notice that they are worlds apart. Hyperion is producing new software, Aeon new hardware, and they have even given the much loved 68k systems a do-over.
This where I get a bit worked up – because Cloanto have nothing to do with software or hardware development. It is quite frankly none of their business (in the true sense of the word). They have licensed the old kernel and Workbench; they have also bought the C64 roms – and that is where their role ends. Yet they spend more time trying to obstruct Hyperion (and by consequence, Aeon) at every step of the way.
While I have no idea who sits on the c64 rights these days, the c64-mini has sold in good numbers around the world. Since Cloanto is the only company with c64 rights I presume they have cashed in on that? Like always it’s hard to tell, because there are more than one company that claim to sit on pieces of the true Commodore legacy.
So to sum up: we have one side producing new hardware, new software and doing updates which is their obligation and right. And we have another party who has created nothing, including the heart of their business, demanding a cut of something they shouldn’t even be involved in (!)
Greed, the mother of invention
Cloanto’s motives should be pretty obvious by now, but let’s hash through it.
With a new Workbench and kernel out in the wild, Cloanto find themselves in a difficult position. Who would want to buy an older kernel or Workbench when there is a newer, 2018 version available? Well, I would like all of them to be honest, but yes I obviously want to use the new versions as much as possible.
The A1222 was due out Q1 2018. It remains on hold until the lawsuits are finished. Keeping Hyperion and Aeon in court is a matter of survival for Cloanto at this point
But that alone is not enough to explain Cloanto’s panic-stricken behavior. They could welcome the new update and simply license it, like they should because they have no right to another companies work.
Instead they run out and buys the remnants of that company I mentioned earlier, Amiga Inc, which is a straw company that has a terrible reputation involving fraud and investor scams. A company that for some magical reason had the right to the name “Amiga” (like that holds any value in 2018, good lord what are you people doing) and sat on the source-code for the OS. This is the same source-code that Hyperion ended up buying, which is no doubt the foundation for the update before xmas.
Why would they go to such lengths as to secure a superficial paper-tiger like Amiga Inc? Trying to reverse the process? Looking to hijack the Amiga names? What gives? It’s almost like Cloanto is looking for something to fight over, desperate to keep Hyperion in court for as long as possible.
And why would they refuse to sell 2000 roms to myself and Gunnar to make ready-to-use Amiga “mini” machines? If I didnt know better, they are brewing on something. The market is just ripe for retro, and their behavior towards us hints that they are not very happy about Amibian’s existence.
It makes even more sense when you factor in the long-awaited A1222. A whole new Amiga that Aeon and Hyperion is 100% invested in bringing to market.
The Amiga A1222 is a Next Generation PPC Amiga that should retail at around USD 450. This product was supposed to reach the market in Q1 2018, but with the lawsuit(s) and drain on funds, getting the product out the door has been impossible. So much so that Cloanto is now damaging Hyperion (and Aeon) by proxy.
Around Xmas 2018 Cloanto began spreading the rumor that they were fighting to “open source Amiga OS”. That is a blatant lie and I was tempted to write a piece there and then, but I have been busy with work. I also thought Amiga users wouldn’t fall for such an evident lie, but some people actually cheer Cloanto on — believing that Cloanto can somehow “help” the Amiga platform. For Christ sake, Cloanto doesn’t even have the source-code – much less the right to open source Hyperion and Acer’s intellectual property. Buying the remnants of Amiga Inc might be an attempt to buy credibility, but its 20 years too late.
The present legalities are, to be blunt, nothing more than a diversion designed to keep the A1222 out of the marketplace. The question is: why and will they try to replace it with something?
Although the motives are now painfully visible, so much so that it might as well be lit up in neon – I think Amiga fans should be very careful where they place their trust. I am sorry but I would not trust Cloanto with a stick of gum, much less the computing legacy of a giant like Commodore. And they are brewing on something, either directly or indirectly, mark my words.
Normally I don’t take sides, but I seriously hope Cloanto wakes up and realize that they are right now, and have been for some time, the spearhead that is keeping the platform in limbo. I have nothing against them personally, but we have now passed the point of no return. You are now risking the codebase of a system that thousands of people care for.
I think I speak for quite a few when I say: Enough! Put that energy, time and money into making something – because whatever you guys started arguing over, is long gone.
There is a whole generation that has grown up without any knowledge of Amiga. Who have no clue what Commodore was and represented. So while you guys have been fighting about who gets to sit where, the boat has left and you missed it.
Final words
You know why I find the most annoying about the situation Cloanto have created? Hear me out here.
Sun Microsystems spent a fortune drumming up support for Java, selling people on a lofty dream where a whole operating-system would be written as bytecodes. And that in special hardware would be made so that bytecodes could run anywhere. Because said bytecodes would be portable between platforms even, and solve the problem with platform bound software once and for all. Companies pumped billions into that dream, yet for all their wealth and power, they failed.
Meanwhile Cloanto, and by extension Hyperion, have had access to UAE since the 90s. A system that embody all the traits that Sun Microsystems attempted to create, and all they have done is to add a menu to it. They have wasted close to two decades without realizing that UAE is that holy grail that Sun Microsystems failed to deliver.
68k machine-code is bytecodes if you execute it on another system. And the distinctions between “virtual machine” and “emulator” are ultimately conceptual – not factual. UAE could have been adjusted as a virtual machine. There you have the compilers, the ecosystem and all the pieces you would need to deliver a portable, blistering fast software deployment system that is truly platform independent.
So, Cloanto, you have been sitting on a gold mine. And you didn’t recognize it because you were too busy arguing over balls, chicken-lip logos, old roms and god knows what else.
You have had solid gold for ages, but you were too busy arguing over names to see it
I sincerely hope Acer takes an active role in their licensing, because as far as I can see, Cloanto is not acting in Acer’s financially best interest (nor Hyperion’s for that matter, which last time I checked can withhold all and any changes to their OS, leaving Cloanto with the dry bones from the past) – and they have become, unless they perform a complete makeover before their next lawsuit, unfit to manage the intellectual property and licenses they have acquired.
You don’t have a developer license, so stick to the legacy stuff and stop getting in the way of those that do.
And for christ sake give the guys who make UAE a percentage, it is tasteless and ugly to watch this level of greed. Seriously.
Quartex Web OS: A cloud OS in takes form
It’s been a while since I’ve posted now. I have 3 articles in escrow, and every time I think I will finish them, I end up writing more. But yes, more Delphi articles is coming and I have lined up both components and rich code that everyone will be happy about.
Please look before shooting
Before we dig into the new stuff, I want to clear up a misconception. We programmers often forget that not everyone knows what we do, and we take it for granted that everyone will instantly understand something we talk about. Which is rarely the case.
I have noticed that quite a few have misjudged the project radically, thinking that the first version (cloud ripper) is just a toy, a mock desktop or even worse: just a remake of a legacy system that “has no role in modern computing”.
It is true that I have taken more than a little from Amiga OS in terms of architecture, but I have exclusively taken ideas that are good and works well under the ASYNC execution model. I have also replicated the way the filesystem is organized, things like REXX (which was added to OS X in 2015), the menu system – these are indeed built on how Amiga OS did things. The same can be said about library functions. Not because they are old, but because they make sense. Many of the functions appear in other systems too, like GTK on Linux and WinAPI for Windows. There are only so many ways to open a window, change the title, define scrollbars and execute processes.
Kiosk systems like this are great targets for the Quartex Web OS
While there are clear architectural aspects taken from older systems, doesn’t mean that the system itself is old in any way. This system is designed to run as WebAssembly, ASM.js and vanilla Javascript – which is ASYNC by nature. It is designed to run and share payload over several machines, not a single outdated CPU and chipset. You have swarm based task solving – which is quite cutting edge if I might say so. None of these things were invented back in the day.
Some have also asked why this is even needed. Well, let me give you a simple use case.
One of my customers is doing work for Jensen, a Danish producer of IT hardware. They make mostly routers, wifi usb dongles and similar devices. But like many hardware vendors their web interface leaves a lot to be desires. Router web interfaces are usually quite annoying and poorly written. Something that should have taken 5 minutes can end up taking 30 just because the design of the interface is rubbish.
With my solution these vendors will be able to drop a whole infrastructure into their products; a infrastructure that provides all the things they need to quickly build a great control panel and router interface. Things like file system mapping, being able to store data to the filesystem through an established websocket protocol; all of it wrapped up in a simple but powerful API. Their settings and features can be represented as programs, which run in windows that are intuitively styled and easy to understand. They will also cut development time dramatically by calling the Quartex Soft-Kernel, rather than having to re-invent everything from scratch.
That is just a tiny, tiny use-case where the desktop and services makes perfect sense. But also keep in mind that the same system can scale up to a 1000 instance Amazon supercomputer if you need to, providing software for your offices and development teams.
In 8 months the desktop is complete (probably before) and I start building the first purely web powered software development toolchain. Everything has been transformed into Javascript (as in compilers, linkers – the whole lot). Both freepascal, clang c/c++ and much more. And developers will be able to login and start producing applications out of the box. The fact that the entire system is chipset and platform independent is quite unique. People tend to use native code behind a facade of html5. Not here. Here you have over 4000 classes, 800.000 lines of code just for the desktop client, looking back at you.
Hopefully this has shed some lights on the project, and people will stop looking at this as “old junk”. As a person who loves older computers, Amiga especially, I am quite frankly astounded by the ignorance regarding that platform. A juiced up 30 year old Amiga will give any modern computer a run for it’s money when it comes to ease of use, quality software and pure productivity. 10 years before Windows even existed, europeans enjoyed a colorful, window based desktop with full multitasking. When we had to switch to PC it was like going back to the 1500’s in terms of functionality – and it wasnt until Windows 7 that Microsoft caught up with Commodore. So if I have managed to get over even 1% of the spirit in that machine – then I will be very happy indeed.
But to limit a clustered, 40 CPU core architecture using modern, off-the-shelves parts, a multitude of node services to “old junk” is nothing short of an intellectual emergency. Please read, digest and look more closely before passing judgement.
Right then, so what’s new?
The Quartex “Cloud Ripper”
Where to begin! Like mentioned in my previous post Amibian.js is a cluster system. As such the project now has its first real hardware sorted! I have gone for a 5 x ODroid XU4 model, neatly tucked inside a PICO 5H case. The budget was set at USD 400, but with shipping and taxes it ended up costing around USD 600. But that is not a bad price for the firepower you get (40 CPU cores, 20 GPU cores and 16 Gb Ram), the ODroid is a powerful, stable and reliable ARM SBC (single board computer). In benchmarks the Raspberry PI 3b scored 830 Dhrystones, the ODroid scored 5500 Dhrystones. And my architecture use five of them, so this is a $600 super-computer built using off the shelves part.
The back-end server has had several bugs fixed, especially the problems with path’s and databases. You can now edit the settings.ini file and tell the system where the database should be created or accessed from, you can set the port for the server, if it should use SSL + Secure WebSocket, or ordinary HTTP + Websocket.
40 ARM CPU cores, that is a lot of firepower for USD 200 !
I am also ditching the TW3NodeFileSystem driver for server logic and using ordinary node.js calls there. The TW3NodeFileSystem driver is mounted as you perform a login – and it acts as a sandbox, mounting your folder as a device (and making sure you can’t ever touch files outside your “home” server folder). We still need to implement a proper UNIX directory parser, but that is easy enough.
Quartex Pascal
Yes, I have picked up Quartex Pascal again, which originally started in 2014. I have started writing a new RTL for DWScript which is an alternative to Smart Mobile Studio. It is different from the Smart RTL and is closer to FMX than VCL.
Eventually the Quartex Web OS and all its services will compile without code from Smart Mobile Studio.
Hosted applications, messages and our soft-kernel
The biggest news, which is also the most tricky to get right, is getting hosted applications (applications are hosted in IFrame containers) to communicate with the desktop. As you probably know browsers have rigid security measures, and the rules for threads (web workers) and separate processes (frames) are severe to say the least.
The LDEF assembler is the first application to grace the system
A secondary application hosted in a frame has absolutely no access to the rest of the DOM. Meaning that the code has no way of calling functions or manipulating elements outside its own DOM in the frame container. This is a good system because we don’t want rouge applications causing havoc.
The only way an application can talk to the desktop is through messages. And while this sounds easy, remember: we are doing this as a solid system, not just slapping something together.
- After loading a hosted application, the desktop will send a handshake request. It will do this on interval until the application accepts.
- When the application replies with a handshake message, the desktop sends a special message-channel object to the app. All communication with the desktop must happen on that secure channel.
- With the channel obtained, the application has to provide the application manifest file. This is a special INI-File containing information about the program, including access rights. None of the soft-kernel API functions will execute until a valid manifest-file has been delivered.
- Once the manifest has been sent and accepted, the hosted application is free to call the soft-kernel functions.
The above might sound simple but it includes several sub technologies to be in place first:
- Call Stack: a class that keep track of sent messages and a callback. When a response arrives it will execute the correct callback to deliver the response. This is a kind of “promises” engine for message delivery.
- Message factory, matches message-data to the correct message class, creates the instance and de-serialize the data automatically for you
- Message dispatcher: Allows you to register a message with a handler procedure. When a message arrives the dispatcher calls the message-factory, then calls the correct handler.
- Base64 Encoding on byte-array, stream and buffer level (does not exist in either node.js or JavaScript in general)
- String to UTF8 Byte-Array encoding
- UTF8 Byte-Array to String encoding
- escape and unescape for byte-array, stream and buffer
- URI-encoder for byte-array, stream and buffer
But that was just the beginning, I also had to introduce an object that I have been dreading to even start on, namely the “process” class. The process is not just a simple reference to the frame container, it has to keep track of the websocket endpoint, application manifest, error handling, message routing and much more.
CLANG compiled to webassembly, meaning we can now compile proper C/C++ in the browser
Since Amibian.js supports not just JavaScript, but also bytecode applications – the process object also contains the LDEF runtime engine; not to mention all the system resources a process can own.
The cool part is that things work exactly like I planned! There is plenty of room to optimize, but all in all the architecture is sound. And it was quite a hallelujah moment when the first API call went through at 00:00 19.01.2019! A call to SetWindowTitle() where the hosted application set the caption of its main-window purely via code. Cross domain communication at it’s very best.
The LDEF Assembler
Yes LDEF Bytecodes are fantastic, and the first program I have made is a traditional assembler. I went all in and implemented a full text-editor to get better control, and also to get rid of the ACE code editor, which was a massive dependency. So glad we got rid of that.
So now you can write assembly code, assemble it, run it, dis-assemble it and even dump the bytecodes to the window. You will be able to save the bytecodes to disk by the end of this weekend, and then run the bytecode programs from shell or the desktop. So we are really making progress here.
A good shell / pipe infrastructure is the key to a powerful desktop
LDEF is the bytecode system that will be used to build high-level languages like Basic and Pascal. Since Freepascal is now able to compile itself to JavaScript I will naturally add that to the IDE next fall; the same is true for CLANG which has compiled itself to WebAssembly — and who generates webassembly.
So C/C++ and object pascal are already working and waiting for the IDE.
LDEF is a grander system though, because libraries can be loaded by Delphi, C++ builder, C# or whatever you fancy – and used. It can be post-processed to real machine code, or converted to pure WebAssembly. It holds much wider scope than stack machines like CLR and Java, and its more natural for assembly programmers – because it’s based on real CPU’s. It’s a register based virtual machine, not a stack-machine.
More?
Tons, but you have to visit my patreon page to keep track. I try to publish as much as possible there rather than here. I post a bit on both, but the proper channel for Amibian.js (or “Quartex Web OS” as its official name is) will always be Patreon.
The picture viewer now has momentum scrolling in full-mode.
Also, fixed more bugs in the Smart RTL than I can count, and re-made window movement. Window movement now uses the GPU, so they are silky smooth everywhere. Resize will be optimized next, then you can’t really tell it’s not native code at all.
Delphi Component updates
Yes Delphi is also a huge part of the Patreon project, and you will be happy to hear that the form designer (which shares a codebase with the graphics application components) have seen more work!
You can check out some of the changes to the form-designer here:
These changes will be in the january update (end of month) together with all the changes to Amibian.js, HexLicense, Tween library and all the rest 🙂
Cheers!
Amibian.js under the hood
Amibian.js is gaining momentum as more and more developers, embedded systems architects, gamers and retro computer enthusiasts discover the project. And I have to admit I’m pretty stoked about what we are building here myself!
In a life-preserver no less 😀
But, with any new technology or invention there are two common traps that people can fall into: The first trap is to gravely underestimate a technology. JavaScript certainly invites this, because only a decade ago the language was little more than a toy. Since then JavaScript have evolved to become the most widely adopted programming language in the world, and runtime engines like Google’s V8 runs JavaScript almost as fast as compiled binary code (“native” means machine code, like that produced by a C/C++ compiler, Pascal compiler or anything else that produces programs that run under Linux or Windows).
It takes some adjustments, especially for traditional programmers that havent paid attention to where browsers have gone – but long gone are the days of interpreted JavaScript. Modern JavaScript is first parsed, tokenized and compiled to bytecodes. These bytecodes are then JIT compiled (“just in time”, which means the compilation takes place inside the browser) to real machine-code using state of the art techniques (LLVM). So the JavaScript of 2018 is by no means the JavaScript of 2008.
The second trap you can fall into – is to exaggerate what a new technology can do, and attach abilities and expectations to a product that simply cannot be delivered. It is very important to me that people don’t fall into either trap, and that everyone is informed about what Amibian.js actually is and can deliver – but also what it wont deliver. Rome was not built-in a day, and it’s wise to study all the factors before passing judgement.
I have been truly fortunate that people support the project financially via Patreon, and as such I feel it’s my duty to document and explain as much as possible. I am a programmer and I often forget that not everyone understands what I’m talking about. We are all human and make mistakes.
Hopefully this post will paint a clearer picture of Amibian.js and what we are building here. The project is divided into two phases: first to finish Amibian.js itself, and secondly to write a Visual Studio clone that runs purely in the browser. Since it’s easy to mix these things up, I’m underlining this easy – just in case.
What the heck is Amibian.js?
Amibian.js is a group of services and libraries that combined creates a portable operating-system that renders to HTML5. A system that was written using readily available web technology, and designed to deliver advanced desktop functionality to web applications.
The services that make up Amibian.js was designed to piggyback on a thin Linux crust, where Linux deals with the hardware, drivers and the nitty-gritty we take for granted. There is no point trying to write a better kernel in 2018, because you are never going to catch up with Linus Torvalds. It’s must more interesting to push modern web technology to the absolute limits, and build a system that is truly portable and distributed.
Above: Amibian.js is created in Smart Pascal and compiled to JavaScript
The service layer is written purely in node.js (JavaScript) which guarantees the same behavior regardless of host platform. One of the benefits of using off-the-shelves web technology is that you can physically copy the whole system from one machine to the other without any changes. So if you have a running Amibian.js system on your x86 PC, and copy all the files to an ARM computer – you dont even have to recompile the system. Just fire up the services and you are back in the game.
Now before you dismiss this as “yet another web mockup” please remember what I said about JavaScript: the JavaScript in 2018 is not the JavaScript of 2008. No other language on the planet has seen as much development as JavaScript, and it has evolved from a “browser toy” – into the most important programming language of our time.
So Amibian.js is not some skin-deep mockup of a desktop (lord knows there are plenty of those online). It implements advanced technologies such as remote filesystem mapping, an object-oriented message protocol (Ragnarok), RPCS (remote procedure call invocation stack), video codec capabilities and much more — all of it done with JavaScript.
In fact, one of the demos that Amibian.js ships with is Quake III recompiled to JavaScript. It delivers 120 fps flawlessly (browser is limited to 60 fps) and makes full use of standard browser technologies (WebGL).
Click on picture above to watch Amibian.js in action on YouTube
So indeed, the JavaScript we are talking about here is cutting edge. Most of Amibian.js is compiled as “Asm.js” which means that the V8 runtime (the code that runs JavaScript inside the browser, or as a program under node.js) will JIT compile it to highly efficient machine-code.
Which is why Amibian.js is able to do things that people imagine impossible!
Ok, but what does Amibian.js consist of?
Amibian.js consists of many parts, but we can divide it into two categories:
- A HTML5 desktop client
- A system server and various child processes
These two categories have the exact same relationship as the X desktop and the Linux kernel. The client connects to the server, invokes procedures to do some work, and then visually represent the response This is identical to how the X desktop calls functions in the kernel or one of the Linux libraries. The difference between the traditional, machine code based OS and our web variation, is that our version doesn’t have to care about the hardware. We can also assign many different roles to Ambian.js (more about that later).
Enjoying other cloud applications is easy with Amibian.js, here is Plex, a system very much based on the same ideas as Amibian.js
And for the record: I’m trying to avoid a bare-metal OS, otherwise I would have written the system using a native programming language like C or Object-Pascal. So I am not using JavaScript because I lack skill in native languages, I am using JavaScript because native code is not relevant for the tasks Amibian.js solves. If I used a native back-end I could have finished this in a couple of months, but a native server would be unable to replicate itself between cloud instances because chipset and CPU would be determining factors.
The Amibian.js server is not a single program. The back-end for Amibian.js consists of several service applications (daemons on Linux) that each deliver specific features. The combined functionality of these services make up “the amibian kernel” in our analogy with Linux. You can think of these services as the library files in a traditional system, and programs that are written for Amibian.js can call on these to a wide range of tasks. It can be as simple as reading a file, or as complex as registering a new user or requesting admin rights.
The greatest strength of Amibian.js is that it’s designed to run clustered, using as many CPU cores as possible. It’s also designed to scale, meaning that it will replicate itself and divide the work between different instances. This is where things get’s interesting, because an Amibian.js cluster doesn’t need the latest and coolest hardware to deliver good performance. You can build a cluster of old PC’s in your office, or a handful of embedded boards (ODroid XU4, Raspberry PI’s and Tinkerboard are brilliant candidates).
But why Amibian.js? Why not just stick with Linux?
That is a fair question, and this is where the roles I mentioned above comes in.
As a software developer many of my customers work with embedded devices and kiosk systems. You have companies that produce routers and set-top boxes, NAS boxes of various complexity, ticket systems for trains and busses; and all of them end up having to solve the same needs.
What each of these manufacturers have in common, is the need for a web desktop system that can be adapted for a specific program. Any idiot can write a web application, but when you need safe access to the filesystem, unified API’s that can delegate signals to Amazon, Azure or your company server, things suddenly get’s more complicated. And even when you have all of that, you still need a rock solid application model suitable for distributed computing. You might have 1 ticket booth, or 10.000 nation wide. There are no systems available that is designed to deal with web-technology on that scale. Yet 😉
Let’s look at a couple of real-life scenarios that I have encountered, I’m confident you will recognize a common need. So here are some roles that Amibian.js can assume and help deliver a solution rapidly. It also gives you some ideas of the economic possibilities.
Updated: Please note that we are talking javascript here, not native code. There are a lot of native solutions out there, but the whole point here is to forget about CPU, chipset and target and have a system floating on top of whatever is beneath.
- When you want to change some settings on your router – you login to your router. It contains a small apache server (or something similar) and you do all your maintenance via that web interface. This web interface is typically skin-deep, annoying to work with and a pain for developers to update since it’s connected to a native apache module which is 100% dependent on the firmware. Each vendor end up re-inventing the wheel over and over again.
- When you visit a large museum notice the displays. A museum needs to display multimedia, preferably on touch capable devices, throughout the different exhibits. The cost of having a developer create native applications that displays the media, plays the movies and gives visual feedback is astronomical. Which is why most museums adopt web technology to handle media presentation and interaction. Again they re-invent the wheel with varying degree of success.
- Hotels have more or less the exact same need but on a smaller scale, especially the larger hotels where the lobby have information booths, and each room displays a web interface via the TV.
- Shopping malls face the same challenge, and depending on the size they can need anything from a single to a hundred nodes.
- Schools and education spend millions on training software and programming languages every year. Amibian.js can deliver both and the schools would only pay for maintenance and adaptation – the product itself is free. Kids get the benefit of learning traditional languages and enjoying instant visual feedback! They can learn Basic, Pascal, JavaScript and C. I firmly believe that the classical languages will help make them better programmers as they evolve.
You are probably starting to see the common denominator here?
They all need a web-based desktop system, one that can run complex HTML5 based media applications and give them the same depth as a native operating-system; Which is pretty hard to achieve with JavaScript alone.
Amibian.js provides a rich foundation of more than 4000 classes that developers can use to write large, complex and media rich applications (see Smart Mobile Studio below). Just like Linux and Windows provides a wealth of libraries and features for native application development – Amibian.js aims to provide the same for cloud and embedded systems.
And as the name implies, it has roots in the past with the machine that defined multimedia, namely the Commodore Amiga. So the relation is more than just visually, Amibian.js uses the same system architecture – because we believe it’s one of the best systems ever designed.
If JavaScript is so poor, why should we trust you to deliver so much?
First of all I’m not selling anything. It’s not like this project is something that is going to make me a ton of cash. I ask for support during the development period because I want to allocate proper time for it, but when done Amibian.js will be free for everyone (LGPL). And I’m also writing it because it’s something that I need and that I havent seen anywhere else. I think you have to write software for yourself, otherwise the quality wont be there.
Secondly, writing Amibian.js in raw JavaScript with the same amount of functions and depth would take years. The reason I am able to deliver so much functionality quickly, is because I use a compiler system called Smart Mobile Studio. This saves months and years of development time, and I can use all the benefits of OOP.
Prior to starting the Amibian.js project, I spent roughly 9 years creating Smart Mobile Studio. Smart is not a solo project, many individuals have been involved – and the product provides a compiler, IDE (editor and tools), and a vast run-time library of pre-made classes (roughly 4000 ready to use classes, or building-blocks).
Writing large-scale node.js services in Smart is easy, fun and powerful!
Unlike other development systems, Smart Mobile Studio compiles to JavaScript rather than machine-code. We have spent a great deal of time making sure we could use proper OOP (object-oriented programming), and we have spent more than 3 years perfecting a visual application framework with the same depth as the VCL or FMX (the core visual frameworks for C++ builder and Delphi).
The result is that I can knock out a large application that a normal JavaScript coder would spend weeks on – in a single day.
Smart Mobile Studio uses the object-pascal language, a dialect which is roughly 70% compatible with Delphi. Delphi is exceptionally well suited for writing large, data driven applications. It also thrives for embedded systems and low-level system services. In short: it’s a lot easier to maintain 50.000 lines of object pascal code, than 500.000 lines of JavaScript code.
Amibian.js, both the service layer and the visual HTML5 client application, is written completely using Smart Mobile Studio. This gives me as the core developer of both systems a huge advantage (who knows it better than the designer right?). I also get to write code that is truly OOP (classes, inheritance, interfaces, virtual and abstract methods, partial classes etc), because our compiler crafts something called a VMT (virtual method table) in JavaScript.
Traditional JavaScript doesn’t have OOP, it has something called prototypes. With Smart Pascal I get to bring in code from the object-pascal community, components and libraries written in Delphi or Freepascal – which range in the hundreds of thousands. Delphi alone has a massive library of code to pick from, it’s been a popular toolkit for ages (C is 3 years older than pascal).
But how would I use Amibian.js? Do I install it or what?
Amibian.js can be setup and used in 4 different ways:
- As a true desktop, booting straight into Amibian.js in full-screen
- As a cloud service, accessing it through any modern browser
- As a NAS or Kiosk front-end
- As a local system on your existing OS, a batch script will fire it up and you can use your browser to access it on https://127.0.0.1:8090
So the short answer is yes, you install it. But it’s the same as installing Chrome OS. It’s not like an application you just install on your Linux, Windows or OSX box. The whole point of Amibian.js is to have a platform independent, chipset agnostic system. Something that doesn’t care if you using ARM, x86, PPC or Mips as your CPU of preference. Developers will no doubt install it on their existing machines, Amibian.js is non-intrusive and does not affect or touch files outside its own eco-system.
But the average non-programmer will most likely setup a dedicated machine (or several) or just deploy it on their home NAS.
The first way of enjoying Amibian.js is to install it on a PC or ARM device. A disk image will be provided for supporters so they can get up and running ASAP. This disk image will be based on a thin Linux setup, just enough to get all the drivers going (but no X desktop!). It will start all the node.js services and finally enter a full-screen web display (based on Chromium Embedded) that renders the desktop. This is the method most users will prefer to work with Amibian.js.
The second way is to use it as a cloud service. You install Amibian.js like mentioned above, but you do so on Amazon or Azure. That way you can login to your desktop using nothing but a web browser. This is a very cost-effective way of enjoying Amibian.js since renting a virtual instance is affordable and storage is abundant.
The third option is for developers. Amibian.js is a desktop system, which means it’s designed to host more elaborate applications. Where you would normally just embed an external website into an IFrame, but Amibian.js is not that primitive. Hosting external applications requires you to write a security manifest file, but more importantly: the application must interface with the desktop through the window’s message-port. This is a special object that is sent to the application as a hand-shake, and the only way for the application to access things like the file-system and server-side functionality, is via this message-port.
Calling “kernel” level functions from a hosted application is done purely via the message-port mentioned above. The actual message data is JSON and must conform to the Ragnarok client protocol specification. This is not as difficult as it might sound, but Amibian.js takes security very seriously – so applications trying to cause damage will be promptly shut down.
You mention hosted applications, do you mean websites?
Both yes and no: Amibian.js supports 3 types of applications:
- Ordinary HTML5/JS based applications, or “websites” as many would call them. But like I talked about above they have to establish a dialog with the desktop before they can do anything useful.
- Hybrid applications where half is installed as a node.js service, and the other half is served as a normal HTML5 app. This is the coolest program model, and developers essentially write both a server and a client – and then deploy it as a single package.
- LDEF compiled bytecode applications, a 68k inspired assembly language that is JIT compiled by the browser (commonly called “asm.js”) and runs extremely fast. The LDEF virtual machine is a sub-project in Amibian.js
The latter option, bytecodes, is a bit like Java. A part of the Amibian.js project is a compiler and runtime system called LDEF.
Above: The Amibian.js LDEF assembler, here listing opcodes + disassembling a method
The first part of the Amibian.js project is to establish the desktop and back-end services. The second part of the project is to create the worlds first cloud based development platform. A full Visual Studio clone if you like, that allows anyone to write cloud, mobile and native applications directly via the browser (!)
Several languages are supported by LDEF, and you can write programs in Object Pascal, Basic and C. The Basic dialect is especially fun to work with, since it’s a re-implementation of BlitzBasic (with a lot of added extras). Amiga developers will no doubt remember BlitzBasic, it was used to create some great games back in the 80s and 90s. It’s well suited for games and multimedia programming and above all – very easy to learn.
More advanced developers can enjoy Object Pascal (read: Delphi) or a sub-set of C/C++.
And please note: This IDE is designed for large-scale applications, not simple snippets. The ultimate goal of Amibian.js is to move the entire development cycle to the cloud and away from the desktop. With Amibian.js you can write a cool “app” in BlitzBasic, run it right in the browser — or compile it server-side and deploy it to your Android Phone as a real, natively compiled application.
So any notion of a “mock desktop for HTML” should be firmly put to the side. I am not playing around with this product and the stakes are very real.
But why don’t you just use ChromeOS?
There are many reasons, but the most important one is chipset independence. Chrome OS is a native system, meaning that it’s core services are written in C/C++ and compiled to machine code. The fundamental principle of Amibian.js is to be 100% platform agnostic, and “no native code allowed”. This is why the entire back-end and service layer is targeting node.js. This ensures the same behavior regardless of processor or host system (Linux being the default host).
Node.js has the benefit of being 100% platform independent. You will find node.js for ARM, x86, Mips and PPC. This means you can take advantage of whatever hardware is available. You can even recycle older computers that have lost mainstream support, and use them to run Amibian.js.
A second reason is: Chrome OS might be free, but it’s only as open as Google want it to be. ChromeOS is not just something you pick up and start altering. It’s dependence on native programming languages, compiler toolchains and a huge set of libraries makes it extremely niche. It also shields you utterly from the interesting parts, namely the back-end services. It’s quite frankly boring and too boxed in for any practical use; except for Google and it’s technology partners that is.
I wanted a system that I could move around, that could run in the cloud, on cheap SBC’s. A system that could scale from handling 10 users to 1000 users – a system that supports clustering and can be installed on multiple machines in a swarm.
A system that anyone with JavaScript knowledge can use to create new and exciting systems, that can be easily expanded and serve as a foundation for rich media applications.
What is this Amiga stuff, isn’t that an ancient machine?
In computing terms yes, but so is Unix. Old doesn’t automatically mean bad, it actually means that it’s adapted and survived challenges beyond its initial design. While most of us remember the Amiga for its games, I remember it mainly for its elegant and powerful operating-system. A system so flexible that it’s still in use around the world – 33 years after the machine hit the market. That is quite an achievement.
The original Amiga OS, not bad for a 33-year-old OS! It was and continues to be way ahead of everyone else. A testament to the creativity of its authors
Amibian.js as the name implies, borrows architectural elements en-mass from Amiga OS. Quite simply because the way Amiga OS is organized and the way you approach computing on the Amiga is brilliant. Amiga OS is much more intuitive and easier to understand than Linux and Windows. It’s a system that you could learn how to use fully with just a couple of days exploring; and no manuals.
But the similarities are not just visual or architectural. Remember I wrote that hosted applications can access and use the Amibian.js services? These services implement as much of the original ROM Kernel functions as possible. Naturally I can’t port all of it, because it’s not really relevant for Amibian.js. Things like device-drivers serve little purpose for Amibian.js, because Amibian.js talks to node.js, and node talks to the actual system, which in turn handles hardware devices. But the way you would create windows, visual controls, bind events and create a modern, event-driven application has been preserved to the best of my ability.
But how does this thing boot? I thought you said server?
If you have setup a dedicated machine with Amibian.js then the boot sequence is the same as Linux, except that the node.js services are executed as background processes (daemons or services as they are called), the core server is initialized, and then a full-screen HTML5 view is set up that shows the desktop.
But that is just for starting the system. Your personal boot sequence which deals with your account, your preferences and adaptations – that boots when you login to the system.
When you login to your Amibian.js account, no matter if it’s just locally on a single PC, a distributed cluster, or via the browser into your cloud account — several things happen:
- The client (web-page if you like) connects to the server using WebSocket
- Login is validated by the server
- The client starts loading preferences files via the mapped filesystem, and then applies these to the desktop.
- A startup-sequence script file is loaded from your account, and then executed. The shell-script runtime engine is built into the client, as is REXX execution.
- The startup-script will setup configurations, create symbolic links (assigns), mount external devices (dropbox, google drive, ftp locations and so on)
- When finished the programs in the ~/WbStartup folder are started. These can be both visual and non-visual.
As you can see Amibian.js is not a mockup or “fake” desktop. It implements all the advanced features you expect from a “real” desktop. The filesystem mapping is especially advanced, where file-data is loaded via special drivers; drivers that act as a bridge between a storage service (a harddisk, a network share, a FTP host, Dropbox or whatever) and the desktop. Developers can add as many of these drivers as they want. If they have their own homebrew storage system on their existing servers, they can implement a driver for it. This ensures that Amibian.js can access any storage device, as long as the driver conforms to the driver standard.
In short, you can create, delete, move and copy files between these devices just like you do on Windows, OSX or the Linux desktop. And hosted applications that run inside their own window can likewise request access to these drivers and work with the filesystem (and much more!).
Wow this is bigger than I thought, but what is this emulation I hear about? Can Amibian.js really run actual programs?
Amibian.js has a JavaScript port of UAE (Unix Amiga Emulator). This is a fork of SAE (scripted Amiga Emulator) that has been heavily optimized for web. Not only is it written in JavaScript, it performs brilliantly and thus allows us to boot into a real Amiga system. So if you have some floppy-images with a game you love, that will run just fine in the browser. I even booted a 2 gigabyte harddisk image 🙂
But Amiga emulation is just the beginning. More and more emulators are ported to JavaScript; you have Nes, SNes, N64, PSX I & II, Sega Megadrive and even a NEO GEO port. So playing your favorite console games right in the browser is pretty straight forward!
But the really interesting part is probably QEmu. This allows you to run x86 instances directly in the browser too. You can boot up in Windows 7 or Ubuntu inside an Amibian.js window if you like. Perhaps not practical (at this point) but it shows some of the potential of the system.
I have been experimenting with a distributed emulation system, where the emulation is executed server-side, and only the graphics and sound is streamed back to the Amibian.js client in real-time. This has been possible for years via Apache Guacamole, but doing it in raw JS is more fitting with our philosophy: no native code!
I heard something about clustering, what the heck is that?
Remember I wrote about the services that Amibian.js has? Those that act almost like libraries on a physical computer? Well, these services don’t have to be on the same machine — you can place them on separate machines and thus its able to work faster.
Above: The official Amibian.js cluster, 4 x ODroid XU4s SBC’s in a micro-rack
A cluster is typically several computers connected together, with the sole purpose of having more CPU cores to divide the work on. The cool thing about Amibian.js is that it doesn’t care about the underlying CPU. As long as node.js is available it will happily run whatever service you like – with the same behavior and result.
The official Amibian.js cluster consists of 5 ODroid XU4/S SBC (single board computers). Four of these are so-called “headless” computers, meaning that they don’t have a HDMI port – and they are designed to be logged into and software setup via SSH or similar tools. The last machine is a ODroid XU4 with a HDMI out port, which serves as “the master”.
The architecture is quite simple: We allocate one whole SBC for a single service, and allow the service to copy itself to use all the CPU cores available (each SBC has 8 CPU cores). With this architecture the machine that deals with the desktop clients don’t have to do all the grunt work. It will accept tasks from the user and hosted applications, and then delegate the tasks between the 4 other machines.
Note: The number of SBC’s is not fixed. Depending on your use you might not need more than a single SBC in your home setup, or perhaps two. I have started with 5 because I want each part of the architecture to have as much CPU power as possible. So the first “official” Amibian.js setup is a 40 core monster shipping at around $250.
But like mentioned, you don’t have to buy this to use Amibian.js. You can install it on a single spare X86 PC you have, or daisy chain a couple of older PC’s on a switch for the same result.
Why Headless? Don’t you need a GPU?
The headless SBC’s in the initial design all have GPU (graphical processing unit) as well as audio capabilities. What they lack is GPIO pins and 3 additional USB ports. So each of the nodes on our cluster can handle graphics at blistering speed — but that is ultimately not their task. They serve more as compute modules that will be given tasks to finish quickly, while the main machine deals with users, sessions, traffic and security.
The 40 core cluster I use has more computing power than northern europe had in the early 80s, that’s something to think about. And the pricetag is under $300 (!). I dont know about you but I always wanted a proper mainframe, a distributed computing platform that you can login to and that can perform large tasks while I do something else. This is as close as I can get on a limited budget, yet I find the limitations thrilling and fun!
Part of the reason I have opted for a clustered design has to do with future development. While UAE.js is brilliant to emulate an Amiga directly in the browser – a more interesting design is to decouple the emulation from the output. In other words, run the emulation at full speed server-side, and just stream the display and sounds back to the Amibian.js display. This would ensure that emulation, of any platform, runs as fast as possible, makes use of multi-processing (read: multi threading) and fully utilize the network bandwidth within the design (the cluster runs on its own switch, separate from the outside world-wide-web).
I am also very interested in distributed computing, where we split up a program and run each part on different cores. This is a topic I want to investigate further when Amibian.js is completed. It would no doubt require a re-design of the LDEF bytecode system, but this something to research later.
Will Amibian.js replace my Windows box?
That depends completely on what you use Windows for. The goal is to create a self-sustaining system. For retro computing, emulation and writing cool applications Amibian.js will be awesome. But Rome was not built-in a day, so it’s wise to be patient and approach Amibian.js like you would Chrome OS. Some tasks are better suited for native systems like Linux, but more and more tasks will run just fine on a cloud desktop like Amibian.js.
Until the IDE and compilers are in place after phase two, the system will be more like an embedded OS. But when the LDEF compiler and IDE is in place, then people will start using it en-mass and produce applications for it. It’s always a bit of work to reach that point and create critical mass.
Object Pascal is awesome, but modern, native development systems are quite demanding
My personal need has to do with development. Some of the languages I use installs gigabytes onto my PC and you need a full laptop to access them. I love Amibian.js because I will be able to work anywhere in the world, as long as a browser and normal internet line is available. In my case I can install a native compiler on one of the nodes in the cluster, and have LDEF emit compatible code; voila, you can build app-store ready applications from within a browser environment.
I also love that I can set-up a dedicated platform that runs legacy applications, games – and that I can write new applications and services using modern, off the shelve languages. And should a node in the cluster break down, I can just copy the whole system over to a new, affordable SBC and keep going. No super expensive hardware to order, no absurd hosting fees, and finally a system that we all can shape and use in a plethora of systems. From a fully fledged desktop to a super advanced NAS or Router that use Amibian.js to give it’s customers a fantastic experience.
And yes, I get to re-create the wonderful reality of Amiga OS without the absurd egoism that dominates the Amiga owners to this day. I don’t even know where to begin with the present license holders – and I am so sick of the drama that rolling my own seemed the only reasonable path forward.
Well — I hope this helps clear up any misconceptions about Amibian.js, and that you find this as interesting as I do. As more and more services are pushed cloud-side, the more relevant Amibian.js will become. It is perfect as a foundation for large-scale applications, embedded systems — and indeed, as a solo platform running on embedded devices!
I cant wait to finish the services and cluster this sucker on the ODroid rack!
If you find this project interesting, head over to my Patreon website and get involved! I could really use your support, even if it’s just a $5 “high five”. Visit the project at: http://www.patreon.com/quartexNow
Jon L. Aasenden 
You must be logged in to post a comment.