Archive for May, 2018

Smart Pascal file enumeration under node.js

May 10, 2018 Leave a comment

Ok. I admit it. Writing an RTL from scratch has been one of the hardest tasks I have ever undertaken. Thankfully I have not been alone, but since I am the lead developer for the RTL, it naturally falls on me to keep track of the thousands of classes it comprises; how each affect the next, the many inheritance chains and subsequent causality timelines that each namespace represents.

We were the first company in the world to do this, to establish the compiler technology and then author a full RTL on top of that – designed to wrap and run on top of the JavaScript virtual machine. To be blunt, we didn’t have the luxury to looking at what others had done before us. For every challenge we have had to come up with solutions ourselves.

Be that as it may, after seven years we have gotten quite good at framework architecture. So whenever we need to deal with a new runtime environment such as node.js – we  have already built up a lot of experience with async JSVM development, so we are able to absorb and adapt much faster than our competitors.

Digging into a new platform

Whenever I learn a new language, I typically make a little list of “how do I do this?” type questions. It can be simple, like writing text to stdout, or more elaborate like memory mapped files, inheritance model, raw memory access and similar topics.

But one of the questions have always been: how do I enumerate files in a folder?

While this question is trivial at best, it stabs at the heart of the sub structure of any language. On operating systems like Linux a file is not just data on a disk like we are used to from Windows. A file can be a socket, a virtual access point exposed by the kernel, a domain link, a symbolic link or a stream. So my simple question is actually motivated to expose the depth of the language im learning. I then write down whatever topics come up and then research / experiment on them separately.

Node, like the browser, executes code asynchronously. This means that the code you write cannot be blocking (note: node does support synchronous file IO methods, but you really don’t want to use them in a server. They are typically used before the server is started to load preferences files and data).

As you can imagine, this throws conventional coding out the window. Node exposes a single function that returns an array of filenames (array of string), which helps, but it tells you nothing about the files. You don’t get the size, the type, create and modify timestamps – just the names.

To get the information I just mentioned you have to call a function called “fs.stat”. This is a common POSIX filesystem command. But again we face the fact that everything is async, so that “for / next” loop is useless.

Luke Filewalker

In version 3.0 of Smart Mobile Studio our Node.JS namespace (collection of units with code) has been upgraded and expanded considerably. We have thrown out almost all our older dependencies (like utf8.js and base64.js) and implemented these as proper codec classes in Smart Pascal directly.

Our websocket framework has been re-written from scratch. We threw out the now outdated websocket-io and instead use the standard “ws” framework that is the most popular and actively maintained module on NPM.

We have also implemented the same storage-device class that is available in the browser, so that you can write file-io code that works the same both server-side and client-side. The changes are in the hundreds so I wont iterate through them all here, they will be listed in detail on the release-notes document when the time comes.

But what is a server without a fast, reliable way of enumerating files?

Well, here at the Smart Company we use our own products. So when writing servers and node micro-services we face the exact same challenges as our customers would. Our job is to write ready solutions for these problems, so that you don’t have to spend days and weeks re-inventing the wheel.

Enumerating files is handled by the class TNJFileWalker (I was so tempted to call it Luke). This takes care of everything for you, all the nitty-gritty is neatly packed into a single, easy to use class.

Here is an example:


Enumerating files has literally been reduced to childs play

The class also expose the events you would expect, including a filtering event where you can validate if a file should be included in the final result. You can even control the dispatching speed (or delay between item processing) which is helpful for payload balancing. If you have 100 active users all scanning their files at the same time -you probably want to give node the chance to breathe (20ms is a good value).

The interface for the class is equally elegant and easy to understand:


What would you prefer to maintain? 500.000 lines of JavaScript or 20.000 lines of pascal?

Compare that to some of the spaghetti JavaScript developers have to live with just to perform a file-walk and then do a recursive “delete folder”. Sure hope they check for “/” so they don’t kill the filesystem root by accident.

const fs = require('fs');
const path = require('path');

function filewalker(dir, done) {
    let results = [];

    fs.readdir(dir, function(err, list) {
        if (err) return done(err);

        var pending = list.length;

        if (!pending) return done(null, results);

            file = path.resolve(dir, file);

            fs.stat(file, function(err, stat){
                // If directory, execute a recursive call
                if (stat && stat.isDirectory()) {
                    // Add directory to array [comment if you need to remove the directories from the array]

                    filewalker(file, function(err, res){
                        results = results.concat(res);
                        if (!--pending) done(null, results);
                } else {

                    if (!--pending) done(null, results);

function deleteFile(dir, file) {
    return new Promise(function (resolve, reject) {
        var filePath = path.join(dir, file);
        fs.lstat(filePath, function (err, stats) {
            if (err) {
                return reject(err);
            if (stats.isDirectory()) {
            } else {
                fs.unlink(filePath, function (err) {
                    if (err) {
                        return reject(err);

function deleteDirectory(dir) {
    return new Promise(function (resolve, reject) {
        fs.access(dir, function (err) {
            if (err) {
                return reject(err);
            fs.readdir(dir, function (err, files) {
                if (err) {
                    return reject(err);
                Promise.all( (file) {
                    return deleteFile(dir, file);
                })).then(function () {
                    fs.rmdir(dir, function (err) {
                        if (err) {
                            return reject(err);

Writing Smart Pascal Controls, async initialization and the tao pattern

May 7, 2018 Leave a comment

Async programming can take a bit getting used to if you come straight from Delphi or Lazarus. So in this little article I am going to show you an initialization pattern that will help you initialize your custom-controls and forms in way that is reliable.

Object Ready

In 99.9% of the custom-controls you create, you will either inherit directly from an existing control (like TW3Button, TW3EditBox or other traditional visual controls), or directly from TW3CustomControl.

If you have a quick look at the source for the RTL, which we take for granted that you do, you will find that our RTL is very familiar. It is loosely based on the LCL (lazarus component library), VCL (Visual component library) and with a dash of Mono GTK# thrown in for good measure. But while familiar in appearance, it really is a completely new RTL written to deliver the best of what HTML5 / JS has to offer.

One of the more interesting methods of TW3CustomControl is ObjectReady. This is actually introduced further down in the inheritance chain with TW3MovableControl, but most developers want the infrastructure TW3CustomControl delivers – so that will be the focus on the topic today.

In short, ObjectReady is called when your visual control has been created, injected into the DOM and is ready for use.

The common mistake

A common mistake with ObjectReady() is that the ready state somehow covers any child elements you might have created for your control. This is not the case. ObjectReady() is called when the current control is finished with its initialization, and is ready for manipulation.

Just before the ObjectReady() method is called, the csReady flag is added to the ComponentState set (note: if you don’t know what a set is, it’s a bit like an array of enums. Please google “Delphi sets” to investigate this further if you are just starting out).

Checking if a control is ready can be done manually by reading the csReady state from a controls ComponentState. But naturally, that only works if the control has already reached that state. Prior to the ready state the csCreating state is added to ComponentState, this is removed as the initialization completes and the control enters ready state.

The united states of custom-controls

To better understand when the different component states are set and what really happens when you create a visual control, let’s go through the steps one by one.

  • TW3TagObj
    • Ordinary constructor (create) is called
      • csCreating is added to ComponentState
      • DOM element name to be managed is obtained via TW3TagObj.MakeElementTagId()
    • Handle is obtained via TW3TagObj.MakeElementTagObj()
      • csLoading in added to ComponentState
      • A DOM level identifier (name) is assigned to the control
      • ZIndex is calculated and assigned to the control
    • StyleTagObject() method is called for any css adjustments
    • InitializeObject() is called, this is the constructor in our RTL
    • Control instance is registered with the global control tracker
      • csCreating is removed from ComponentState
      • csLoading in removed from ComponentState
  • TW3MovableControl
    • Alpha blending is initialized but not activated
    • if cfIgnoreReadyState is set in CreationFlags() then ObjectReady is called immediately without any delay
    • If not cfIgnoreReadyState is set, the ReadySync() method is called

the ReadySync() method is of special importance here.

Since JavaScript is asynchronous, reality is that whatever child controls you have created during InitializeObject, can still be under construction even after InitializeObject() finishes. The JavaScript engine might have returned a handle, but the data for the instance is still under construction behind the scenes.

To be blunt: Never trust JavaScript to deliver a 100% ready to use element. If the browser is under heavy stress from other websites and internal work, that can have catastrophic consequences on the state of the objects it returns.

This is one of many reasons that made us chose to write our RTL from scratch rather than just fork CLX or try to be as Delphi friendly as possible. That would have been much easier for us, but it would also be to sell you on the tooth-fairy because that’s not how JavaScript works.

We want our users to have full control and enjoy the same freedom and simplicity that made us fall in love with object pascal all those years ago. And if we forced JavaScript into a pre-fabricated mold like the LCL; the spark and flamboyance that JavaScript brings to the table would have been irreparably damaged if not lost.

But let’s get back on topic.

Like mentioned above, ReSync() is of special importance. It will continuously check if the control is “ready” on a 10ms interval, and it keeps going until the criteria matches or it times out. To avoid infinite loops it has a maximum call stack of 300. Meaning it will keep trying 300 times, a total of 3 seconds and then break out free for safety reasons.

But once the criteria for ready-state matches (or the waiting interval times out)  – ObjectReady() is finally called.

Keep your kids in order

While knowing when the control is ready for us is great for writing components, what practical purpose does it really serve if the child controls is excluded?

Well, again we are back at freedom. We could have baked in a wait sequence for our forms (since the designer knows what child elements are involved). But sadly that wont work on custom controls that the IDE doesn’t manage. And it would only work on forms.

A negative side-effect of this (yet I did test it) is that a form will remain blank until all child controls, their children and their grand children – all reports “ready”.

In short: our code cannot manage what it doesn’t know. The IDE cannot (for reasons obvious) know what your code creates at runtime. And in large and complex controls like grids, planners or MDI systems – such code would get in your way and render the benefits null and void quickly.

As of writing there are some creative solutions to this, trying to get the timing right

  • Write your own checking routines inspired by ReadySync
  • Ignore the whole thing and just check ready-state and that child elements are not NIL in Resize(). This is what most people do.
  • Use TW3Dispatch and throw in an extra Resize() call somewhere in ObjectReady()

While perfectly legal (or perhaps better said: not illegal), these solutions are not very reliable. If the browser is under stress it can prioritize your layout code as less important – and suddenly you have a button where it’s not supposed to be, or a panel that never stretched as planned.

The Tao pattern

Tao (time aware operation) is pattern I created to solve this problem with a bit of grace. Much like the ReadySync() method we talked about earlier, it performs interval based checking of child element states, and thus allows you to do operations in a timely fashion.

As you probably know, under Smart Pascal you are not supposed to override the constructor when you create new controls. Instead you override InitializeObject(). The same goes for the destructor, there you override FinalizeObject().

So the 5 “must-know” methods for writing your own controls are:

  1. InitializeObject
  2. FinalizeObject
  3. ObjectReady
  4. Resize
  5. StyleTagObject

Note: Since Smart Mobile Studio has an evolved theme engine, it is rare that people override StyleTagObject() these days. But there are cases where you want to set some HTML attribute or alter a style; changes that are too small to justify a new style in the global stylesheet. It’s also the place to call ThemeReset() if you don’t want your control to use themes at all, or perhaps set a different theme border and background.

OK, let’s look at a practical example of how TAO works. It is simple, flexible and requires minimal adaptation if you want to adjust older controls you have made.


Lets build a simple path selector control. Easy and ad-hoc

In this example we will be making a path selector. This is essentially an edit box with a button situated at the far-right. Clicking the button would bring up some form of dialog. I am excluding that for brevity since it’s not the control that is interesting here, but rather how we initialize the control.


  TTaoControl = class(TW3CustomControl)
    FButton:  TW3Button;
    FEdit:    TW3EditBox;
    procedure InitializeObject; override;
    procedure FinalizeObject; override;
    procedure ObjectReady; override;
    procedure StyleTagObject; override;
    procedure Resize; override;

As you can see the control class is defined exactly the same way as before. There is no change what so ever in how you write your classes. Now let’s look at the implementation:

procedure TTaoControl.InitializeObject;
  TransparentEvents := false;
  SimulateMouseEvents := false;

  // Create our editbox
  FEdit := TW3EditBox.Create(self);
  FEdit.SetSize(80, 28);

  // reate our select button
  FButton := TW3Button.Create(self);
  FButton.SetSize(70, 28);

procedure TTaoControl.FinalizeObject;

procedure TTaoControl.ObjectReady;
  // set some constraints (optional)
  Constraints.MinWidth := 120;
  Constraints.MinHeight := 32;
  Constraints.Enabled := true;

  // TAO: Wait for the child controls to reach ready-state
  TW3Dispatch.WaitFor([FEdit, FButton], 5,
    procedure (Success: boolean)
      if Success then
        // set some properties for the edit box
        FEdit.ReadOnly := true;
        FEdit.PlaceHolder := 'Please selected a file';

        // set caption for button
        FButton.Caption := 'Select';

        // Do an immediate resize

procedure TTaoControl.StyleTagObject;

  // Set a container border. This border is
  // typically used by TW3Panel and TW3GroupBox
  ThemeBorder := btContainerBorder;

procedure TTaoControl.Resize;
  LBounds:  TRect;
  dx, dy: integer;
  wd, EditWidth: integer;
  // Make sure we dont do anything if resize has been
  // called while the control is being destroyed
  if not (csDestroying in ComponentState) then
    // Make sure we have ready state
    if (csReady in ComponentState) then
      // Check that child elements are all assigned
      // and that they have their csReady flag set in
      // ComponentState. This can be taxing. A more lightweight
      // version is TW3Dispatch.Assigned() that doesnt check
      // the ready state (see class declaration for more info)
      if TW3Dispatch.AssignedAndReady([FButton, FEdit]) then
        // Finally: layout the controls. This can be
        // optimized quite a bit, but focus is not on
        // layout code, but rather the sequence in which operations
        // are executed and handled.
        LBounds := ClientRect;
        wd := LBounds.width;
        dy := (LBounds.Height div 2) - (FEdit.Height div 2);
        EditWidth := (wd - FButton.Width) - 4;
        FEdit.SetBounds(LBounds.left, dy, EditWidth, FEdit.Height);

        dx := LBounds.left + EditWidth + 2;
        dy := (LBounds.Height div 2) - (FButton.Height div 2);
        FButton.SetBounds(dx, dy, FButton.Width, FButton.Height);

If you look closely, what we do here is essentially to spread the payload and cost of creating child elements over multiple methods.

We reduce the constructor, InitializeObject(), to just creating our child controls and setting some initial sizes. This last point, setting initial size, is actually important. Because if the control has no size (width = 0, or height = 0) the browser will not treat the element as visible. Which in turn causes TW3Dispatch.WaitFor() to wait until a size is set.

TW3Dispatch methods

TW3Dispatch is a partial class. This is something that neither Delphi or Freepascal supports, and it has it’s root in C# and the dot net framework.

In short it means that a class can have it’s implementation spread over multiple files. So instead of having to complete a class in a single unit, or inherit from a class and then expanding it – partial classes allows you to expand a class over many units.

This is actually really cool and useful, especially when you work with multiple targets. For example, TW3Dispatch is first defined in System.Time.pas which is the universal namespace that only contains code that runs everywhere. This gives you the basic functionality like delayed execution (just open the unit and have a look).

The class is then further expanded in SmartCL.Time (SmartCL being the namespace for visual, HTML based JavaScript applications). There it gains methods like RequestAnimationFrame() which doesnt exist under node.js for example.

Smart Mobile Studio’s namespaces makes good use of partial classes

TW3Dispatch is further expanded in SmartCL.Components.pas, which is the core unit for visual controls. So starting with version 3.0 the functions I have just demonstrated will be available in the RTL itself.

Until then, you can download TW3Dispatch with the TAO methods here. You need to put it in your own unit, and naturally – use it with care.

Click here to download the TW3Dispatch code.

Note: This code is not free or open-source. It is indended for Smart Mobile Studio owners exclusively, and made available here so registered users can start working with the control coding pattern.