Smart Pascal: stop sharing beta code

I was a bit sad to hear that our RTL, despite the testing team being a very closed group of people, had made its way into the hands of others. People who then blog or comment that its unstable or that to many changes has been added.

When the phrase “early beta” and “our first sprint is to test the low-level functionality, like pointers, memory allocation and manipulation, streams, compression, codecs and type conversion” is clearly stated, then why on earth would someone start at the other end (the visual high level) which is due for changes weeks into the future?

Since this is doing the rounds I figured it is only fair that I clear up a few things:

An async run-time-library that should deliver identical behavior on 9 operating systems and a plethora of browsers, is not something you slap together over a weekend. Many of the solutions in SMS are the result of months of hard work; research and testing on both PC, Mac, various mobile devices and six different embedded boards.

I don’t know how many hours I have put into this project these past 6 years, but its more than I care to admit.

Obviously an early beta is not going to give you 100% compatibility. Especially not since the whole point of the first sprint was for our group to write, test and make sure the low-level methods are rock solid before we move on.

The second sprint is due soon, here we will look at databases, data binding, node.js and writing system level services (via the PM2 process manager).

The last sprint would focus on the visual, high level aspect of the RTL. Things like custom controls, layout, non visual components, event objects, messaging and user interface code.

I’m really gutted that we didn’t even make it through the first sprint before the code was shared. I have no idea who did this, and I frankly don’t care. But from now on I will only throw ball with people I trust and have a friendship with.

People doing reviews and publishing opinions on an upgrade that is presently 1/3 into the final release candidate – I just feel that is unfair.

Polyfills are not shims, and they are not dangerous

It was voiced that the new RTL loads in a couple of files that were considered dangerous by google. Actually that cannot be further from the truth.

But stop and think about this: why would I link to external JavaScript files for things like mutation-events and observers? If these things are available in browsers (which they have been for a long time) why would I need external files?

The answer is that these files are polyfills. And there is a huge difference between what is known as a shim, and a polyfill.

A polyfill is a small JavaScript library that checks if the browser supports the features you want. But if it doesn’t – then the polyfill will register its own code to compensate for the lack of a function or feature. In other words a polyfill is a full implementation (!)

So even if Google, Mozilla or Microsoft remove support for node-observers or mutation-events tomorrow, your application wont be affected by it. Because the polyfill libraries are just that: full library implementations with no dependencies.

In the early beta I shipped out both the observer and mutation-events library has referenced. This has already been changed, and unless you include SmartCL.Observer.pas or SmartCL.MutationEvents.pas – these will no longer be linked automatically. That was never the plan either.

Had one of you (I have come across 3 different places with comments and info on the RTL around the web) bothered to just ask me, I would have included you in the beta process and explained this.

The executables have become so large!

This is another statement I stumbled over: “The new RTL is huge and it generates large JavaScript binaries!”.

First of all you were using an early beta, which means things will be in flux while each level of the RTL is being tested, probed and verified. The pascal uses section in some units referece code they no longer use. Cleaning up that is due for sprint 3. But again in our first sprint focus was on memory, binary data, type conversion, streams, compression and various low-level stuff – not high level code!

As for size that is actually wrong: our namespacing and unit fragmentation is there for a reason. By dividing files into 3 namespaces, fragmenting large units with many classes into more files, our linker is able to trim away unused code more efficiently; as well as optimize your code beyond anything people have seen so far.

Rule of thumb: when you have finished writing your application, always switch on the optimization like you see above for the final build. So yes, you are expected to use packing, obfuscation etc. before deployment. Not just to protect your code, but also to make it load and run faster.

Since you will either use your applications on a mobile device, an embedded device or just online like a website (or desktop via nodewebkit compilation) – you can further speed up loading by activating gzip compression on the server. You will be surprised just how fast and small your applications are when you tune them correctly.

Practical example

The Smart Embedded and Cloud desktop consists of roughly 40+ unit files. Since this desktop can emulate older systems, like the 68k Amiga, Atari, Super Nintendo and much more (more about those later) I have made linking these in optional (read: you have to include the units in order to link the code in, UAE.Core.pas being the primary file).

Without the 68k emulation layer (and various other emulators, which is just added for fun and inspiration) and Aros ROM files (the bios file for the emulator), we are looking at code size in the ballpark of 4.7 megabyte. Which is nothing compared to high-end native solutions.

If you include the 68k emulation layer, the end result is a 10 megabyte index.html file. Again this is small potatoes if you work with embedded systems or kiosk booths.

But since the new RTL design was created to be heavily optimized, far better than the older RTL ever was, the end result after compression is 467 kilobyte (!). When you compress the version that includes the 68k emulation layer – it optimizes down to 1.5 megabyte of code.

• No 68k emulation layer = 4.7 megabyte uncompressed
• No 68k emulation, compressed = 457 kilobyte (!)
• 68k emulation added = 10.2 megabyte uncompressed
• 68k emulation included, compressed = 1.5 megabyte

From 4.7 megabyte to 467 kilobyte

This level of compression and optimization was planned.  I have re-arranged the whole RTL in order to create the best possible circumstance for the compression and obfuscations modules – and I believe the results speaks for itself.

Is the new RTL larger? Indeed it is, with plenty of new classes, controls and libraries. The more features you want to enjoy, the more infrastructure must be in place. That is just how it is, regardless of language.

But you should also know that the new and longer inheritance chain has little or no impact on execution speed. This is what the compiler options “de-virtualization” and “smart linking” are all about. In most cases the code for TW3OwnedObject and TW3OwnedErrorObject are linked into TW3Component. Here is the inheritance chain as of writing:

• TW3OwnedObject
  • TW3OwnedErrorObject
    • TW3Component
      • TW3TagObj
        • TW3TagContainer
          • TW3MovableControl
            • TW3CustomControl

Why did I change this? There are many reasons. First of all I wanted to make the RTL more compatible with Delphi, where names like TComponent and TCustomControl are known and understood by most. So when facing TW3Component or TW3CustomControl, most Delphi developers will intuitively know what these are all about. TW3Component is also the base class for non visual, drag & drop components.

  TW3OwnedErrorObject = class(TW3OwnedObject)
  private
    FLastError: string;
    FOptions:   TW3ErrorObjectOptions;
  protected
    property    ErrorOptions: TW3ErrorObjectOptions read FOptions;
    function    GetLastError: string;
    procedure   SetLastErrorF(const ErrorText: string;
                const Values: Array of const);
    procedure   SetLastError(const ErrorText: string); virtual;
    procedure   ClearLastError;
  public
    property    LastError: string read FLastError;
    constructor Create(const AOwner: TObject); override;
    destructor  Destroy; override;
  end;

There are also other benefits, like uniform error handling. While this might seem useless in an exception based language, there are places where an exception is not something you want to throw. Like in a system level service running under node.js.

In such cases you should catch whatever exception occures and call SetLastError() with the description. The caller will then check if something went wrong and can take measures.

Embedded, a different beast

One of the things Smart Pascal does exceptionally well is embedded platforms. Regardless of what you have been hired to do, be it a kiosk booth in a museum, a touch based vending machine, a parking payment terminal or just some awesome IOT project, Smart Pascal makes this childs play. And the reason its so easy to write powerful web applications is precisely due to the broad scope of our RTL.

Smart Pascal is not a tool for creating websites (well you can, but that’s not what it was designed for). It is a tool to write high quality, complex and feature rich applications.

The distinction between a “web application” and “website” may seem subtle at first, but it all boils down to depth. The Smart RTL now contains a huge amount of classes and third party libraries that simplify writing modern applications with depth.

Other frameworks that (for some reason) are very popular, like JQuery, JQuery UI and ExtJS lacks this depth. They look great, but thats it. I recently catched up with a project that had taken the developers a whole year to create in pure JS. It took me 3 weeks to finish what they spent over 12 months implementing. And that has to do with depth and proper inheritance – not that absurd prototype cloning JavaScript coders use.

To sum up: if you want to enjoy the same features and power in the browser as you do with Delphi or Lazarus on your desktop – then you cant expect Smart to produce binaries smaller than it’s native counterparts.

But more importantly, on embedded systems like the Raspberry PI, the UP x86 board or the new and sexy Tinkerboard — do you really think its going to matter if your application is 4, 8 or 10 megabyte? By now you should have realized that our compressor and obfuscator bakes that down to 460kb or 1.5 megabyte — both peanuts compared to similar system written in Java, C# or Delphi.

Ask and ye shall recieve

I dont really care who shared my RTL, but I am sad that it happened. Im not a difficult person when it comes to these things – all you have to do is ask.

Also remember that the RTL is a copyrighted product. It belongs to a registered financial entity (read: company) and is not subject for distribution.