Archive

Archive for the ‘OP4JS’ Category

Smart Pascal: Busting browser storage limits

April 2, 2017 Leave a comment

Sessionstorage is the name for a browser’s in-memory only storage. Meaning that it’s essentially a ram-disk that is just deleted when you navigate away from the website or close the browser.

Sessionstorage has also been deprecated, so you should avoid using it and go for Localstorage, or just use a raw, untyped uint8 array instead.

Or should you?

Ensuring 64 megabytes minimum

Browsers do not behave identically across devices. Try to get a concurrent reading of something as simple as drawing sprites, and you will quickly notice that even the same device families (Android, iOS and Microsoft) can behave differently between versions – and even builds (revisions).

On embedded systems or thin clients with very little memory, allocating large chunks ot uint8 arrays is not going to work. One of my test thin-client machines has only 512 megabyte ram – and it would throw an exception if I tried to allocate more than 20 megabyte of continuous memory (again, as an array of uint8 bytes).

Using the dark side of the force

Screenshot

Offline means the system boots from a local cache disk

While testing Smart code on this little device, I noticed that quite large images loaded just fine. So where I was not allowed to allocate more than 20 megabytes, the browser would happily load in pictures taking up over 50 megabyte of pixel data?

It then struck me that the maximum limit of a picture, which is enforced by the DIB Api (at least on Windows desktop and embedded), is 4000 x 4000 pixels. Since each pixel is 32 bits (4 bytes, RGBA) that my friend is 64 megabytes right there!

I created a new class that inherits from the virtual-filesystem that Smart Pascal uses, created an off-screen image object in the constructor – and then made a simple but effective “bytes to scan line” calculation routine. So whenever the need for more data grew, it would first grow the picture so it could hold the data (and shrink it again) on demand.

Humble but meaningful

Now 64 megabytes might not seem like much in our day and age, but if you are on holiday and want to connect to your home NAS – 64 megabytes of available ram makes a huge difference. Remember that localstorage only allows between 5 and 10 megabytes.

I should mention that using an image as a buffer makes little sense on a full Windows PC, a Mac or a Linux box. These system will page memory to disk and you will most likely never encounter the 20 megabyte barrier I experienced on this low-end Dell thin client device. But considering that hotels, motels and b&b often have thin clients setup for their customers (read: you) – The Smart desktop has to take height for it.

 

 

 

 

 

Amibian + Smart pascal = A new beginning

March 25, 2017 Leave a comment

In the Amiga community there is a sub-group of people with an agenda. They hoard and collect every piece of hardware they can get their hand on – then sell them at absurd prices on ebay to enrich themselves. This is not only ruining the community at large, ensuring that ordinary users cannot get their hands on a plain, vanilla Amiga without having to fork out enough dough to buy a good car.

“We also have a working Facebook clone – but we’re not
going into competition with Mark Zuckerberg either”

Thankfully not everyone is like that. There are some very respected, very good people who buy old machines, restore them – and sell them at affordable prices. People that do this as a hobby or to make a little on the side. Nothing wrong with that. No, its people that try to sell you an A2000 for $3000, or that pimp out Vampire accelerator cards at 700€ a piece thats the problem.

As for the price sharks, well – this has to stop. And Gunnar’s Amibian distro has already given the Amiga scalpers a serious uppercut. Why buy an old Amiga when you can get a high-end A4000 experience with 4 times as much power for around $35? This is what Gunnar has made a reality – and he deserves a medal for his work!

And myself, Thomas and the others in our little band of brothers will pick up the fight and stand by Gunnar in his battle. A battle to make the Amiga affordable for ordinary human beings that love the platform.

Amiga as a service

Yesterday I did a little experiment. You know how Citrix and VMWare services work? In short, they are virtualization application servers. That means that they can create as many instances of Windows or Linux as they want – run your applications on it – and you can connect to your instance and use it. This is a big part of how cloud computing works.

While my little experiment is very humble, I am now streaming a WinUAE instance display from my basement server pc, just some old bucket of chips I use for debugging, directly into Amibian.js (!). It worked! I just created the world’s first Amiga application server. And it took me less than 30 minutes in Delphi !

Amibian, Amibian.js, appserver, what gives?

Let’s clear this up so you dont mix them:

Amibian. This is the original Linux distro made by Gunnar Kristjansson. It boots straight into UAE in full-screen. All it needs is a Raspberry PI, the Amiga rom files and your workbench or hard disk image. This is a purely native (machine code) solution.

Amibian.js – this is a JavaScript remake of AmigaOS that I’m building, with the look and feel of OS 4. It uses uae.js (also called SAE) to run 68k software directly in the browser. It is not a commercial product, but one of many demos that ship with Smart Mobile Studio. “Smart” is a compiler, editor and run-time library sold by The Smart Company AS. So Amibian.js is just a demo, just like Amos shipped with a ton of demo’s and Blitzbasic came with a whole disk full of examples.

Amiga application server (what I mentioned above) was just a 30 minute experiment I did yesterday after work. Again, just for fun.

I hope that clears up any confusion. Amibian.js is a purely JavaScript based desktop made in the spirit of the Amiga – it must not be confused with Amibian the Linux distro, which boots into UAE on a Raspberry PI. Nor is it an appserver – but rather, it can connect to an appserver if I want to.

genamiga

Generation Amiga post on Amibian.js earlier when I added some look & feel

With a bit of work, and if everything works as expected (I don’t see why not), I will upload both source and binaries to github together with Amibian.js.

There is only one clause: It cannot be used, recreated, included or distributed by Cloanto. Sorry guys, but the ROMS belong to the people, and until you release those into public domain, you wont get access to anything we make. Nothing personal, but pimping out roms and even having audacity to fork UAE and sell it as your own? You should be ashamed of yourself.

Are you in competition with FriendOS?

This question has popped up a couple of times the past two weeks. So I want to address that head on.

I make a product called Smart Mobile Studio. I do that with a group of well-known developers, and we have done so for many years now. The preliminary ideas were presented on my blog during the winter 2009, early 2010 and we started working (and blogging) after that. Smart Mobile Studio and it’s language, Smart Pascal (see Wikipedia article), takes object pascal (like freepascal or Delphi) and compiles to JavaScript rather than machine code.

smsamiga

The Smart compiler is due for OS4 once i get the A1222 in my hands

One of the examples that has shipped with Smart Mobile Studio, and also been available through a library called QTX, is something called Quartex Media Desktop. Which is an example of a NAS server front-end, a kiosk front-end (ticket ordering, cash machines etc) or just an intranet desktop where you centralize media and files. It is also node.js powered to deal with the back-end filesystem. This is now called amibian.js.

In other words – it has nothing to do with Friend software labs at all. In fact, I didn’t even know Friend existed until they approached me a few weeks ago.

15590846_10154070198815906_4207939673564686511_o

The Quartex Media desktop has been around for ages

Amibian.js is just an update of the Quartex Media Desktop example. It is not a commercial venture at all, but an example of how productive you can be with Smart pascal.

And it’s just one example out of more than a hundred that showcase different aspects of our run-time library. This example has been available since version 1.2 or 1.3 of Smart, so no, this is not me trying to reverse engineer FriendOS. Because I was doing this long before FriendOS even was presented. I have just added a windowing manager and made it look like OS4, which also happened before I had any contact with my buddies over at Friend Software Labs (why do you think they were interested in me).

16805402_1914020898827741_149245853_o

Early 2017 Linux bootloader by Gunnar

So, am I in competition with Friend? NO! I have absolutely no ambition, aspiration or intent for anything of the sorts. And should you be in doubt then let me break it down for you:

  • Hogne, Arne, David, Thomas, Francois and everyone at Friend Software Labs are friends of mine. I talk almost daily with David Pleasence who is a wonderful person and an inspiration for everyone who knows him.
  • Normal people don’t sneak around stabbing friends in the back. Plain and simple. That is not how I was raised, and such behavior is completely unacceptable.
  • Amibian.js is 110% pure Amiga oriented. The core of it has been a part of Smart for years now, and it has been freely available for anyone on google code and github.
  • For every change we have made to the Smart RTL, the media desktop example has been updated to reflect this. But ultimately it’s just one out of countless examples. We also have a working Facebook clone – but we’re not going into competition with Mark Zuckerberg for that matter.
  • People can invent the same things at the same time. Thats how reality works. There is a natural evolution of ideas, and great minds often think alike.

Why did you call it Amibian.js, it’s so confusing?

Well it’s a long story but to make it short. The first “boot into uae” thing was initially outlined by me (with the help of chips, the UAE4Arm maintainer). But I didn’t do it right because Linux has never really been my thing. So I just posted it on my retro-gaming blog and forgot all about it.

Gunnar picked this up and perfected it. He has worked weeks and months making Amibian into what it is today – together with Thomas, our spanish superhero /slash/ part-time dictator /slash/ minister of propaganda 🙂

We then started talking about making a new system. Not a new UAE, but something new and ground breaking. I proposed Smart Pascal, and we wondered how the Raspberry PI would run JavaScript performance wise. I then spent a couple of hours adding the icon layout grid and the windowing manager to our existing media desktop – and then fired up some HTML5 demos. Gunnar tested them under Chrome on the Raspberry PI — and voila, Amibian.js was born.

amidesk

And that is all there is to it. No drama, no hidden agendas – and no conspiracy.

I should also add that I do not work at Friend Software Labs, but we have excellent communication and I’m sure we will combine our forces on more than one software title in the future.

On a personal note I have more than a few titles I would like to port to FriendOS. One of my best sellers of all time is an invoice and credit application – which will be re-written in Smart Pascal (its presently a mix of Delphi and C++ builder code). The same program is also due to Amiga OS 4.1 whenever I get my A1222 (looking at you Trevor *smile*).

Well, I hope that clears up any misunderstanding regarding these very separate but superficially related topics. Amibian.js will remain 100% Amiga focused – that has been and remains our goal.

Ode to our childhood

Amibian is and will always be, an ode to the people who gave us such a great childhood. People like David Pleasence who was the face of Commodore in europe. A man who embody the friendliness of the Amiga with his very being. Probably one of the warmest and kindest people I can think of.

Francois Lionet, author of Amos Basic. The man who made me a programmer and that I cannot thank enough. And I know I’m not alone about learning from him.

Mark Sibly, the author of BlitzBasic, the man who taught me all those assembler tricks. A man that deserves to go down in the history books as one of the best programmers in history.

And above all – the people who made the Amiga itself; giants like Jay Miner, Dave Haynie, Carl Sassenrath, Dave Needle, RJ Michal (forgive me for not listing all of you. Your contributions will never be forgotten).

That is what Amibian.js is all about.

Patents and greed may have killed the actual code. But we are free to implement whatever we like from scratch. And when I’m done – your patents will be worthless..

 

Amiga revival, Smart Pascal and growing up

March 11, 2017 1 comment

Maybe its just me but the Amiga is kinda having a revival these days? Seems to me like the number of people going back to the Amiga has just exploded the past couple of years. Much of that is no doubt thanks to my buddy Gunnar Kristjannsen’s excellent work on the Amibian distro for Raspberry PI. Making a high-end Amiga experience that would have cost you thousands of dollars available at around $35.

amifuture

Looking forward to some cosy reading

While Gunnar’s great distro is no doubt a huge factor in this, I believe its more than just easy access. I think a lot of us that grew up with the system, who lived the Amiga daily from elementary school all the way to college – have come full circle. We spend our days coding on PC’s, Mac’s or making mobile software – but deep down inside, I think all of us are still in love with that magical machine; The Commodore Amiga.

I am honestly at a loss for words on this (and that’s a first, most days you can’t get me to shut the hell up). Why should a 30-year-old system attract me more, and still cause so much joy in my life – compared to the latest stuff? I mean, I got a fat ass I7 that growls when you start it with 64 gigabyte ram, SSD and all the extras; I got macs all over the house, the latest consoles – and enough embedded boards to start my own arcade if I so desired.

Yet at the end of the day, when the kids are in bed and GF firmly planted in front of her favorite tv show, fathers are down in basements all around europe. Not watching porn, not practising black magic or trying to transform led into gold, nope: coding in assembler on a mc68k processor running at a whopping 7Mhz and loving every minute of it!

Today the madness held no bounds and forced me, out of sheer perverted joy, to order 4 copies of Amiga Future magazine (yes there are still magazines for the Amiga, believe it or not), a few posters, a mousemat and (drumroll) the ever sexy A1222. Actually that was a lie, I ordered that weeks ago, Trevor Dickenson over at A-EON hooked me up so im getting it as soon as it comes off the assembly line. And for those that don’t know, the A1222 is the new affordable Amiga that is released today. It’s not a remake of the older models, but a brand new thing. I havent been this giddy about a piece of silicon since I fell into a double-d cup at a beach in Spain last year.

Smart Pascal

It made sense to unite my two great computing passions, namely the object pascal language and Amiga into one package. So whenever I have some spare time I work my ass off on the update for Smart Mobile Studio. And it’s getting probably the biggest “demo” ever shipped with a programming language.

What? Well, a remake of the Amiga operating system. But not just a simple css-styled shallow lookalike. You know me, I just had to go all the way. So I married the system with something called uae.js. Which is essentially the JavaScript version of the Amiga emulator. Its compiled with EmScripten – a post processor that takes LLVM compiled bitcode compiled with C/C++ and spits out Asm.js optimized code.

amidesk

You just cant kill it, Amiga is 4ever

So, Smart Pascal in one hand – C/C++ in the right hand. Its like being back in college all over again. Only thing missing now is that Wacom suddenly returns and Borland rise from the grave with another Turbo product. But yes, JavaScript is something I really enjoy. And being able to compile object pascal to JavaScript is even better.

The end result? Well since I don’t have too much time on my hands it’s roughly 31-32% done, and when we hit 50% is when UAE.js will be activated. So right now its a sexy cloud front end. It has a virtual filesystem that runs fine over localstorage, but it can also talk to node.js and access the real filesystem on your server.

But when UAE.js kicks in you will be able to run your favorite Amiga demos, applications and games in your browser. I am actually very excited about seeing the performance. It runs most demos OK (using the Aros rom-files). I imagine running things like blitzbasic, Amos basic and SAS-C/C++ should work fine. Or at least be within the “usable” range if you got a powerful PC to play with.

The V8 JavaScript engine in webkit is due for an overhaul next year – and while I can only speculate I’m guessing real-life compilation will be the addition. They already do some heavy JIT’ing but once you throw LLVM based actual compilation into the picture – large JS applications is going to fly side by side with native stuff. And that’s when cloud front-ends like ChromeOS and other FriendOS is going to take off.

My little remake is not that ambitious, but I do intend to make this an absolute kick-ass system as far as Amiga is concerned. And for Smart Pascal developers? Well, lets just say that this demo project has pushed the RTL for all it’s worth and helped fix bugs and expand the RTL in a way that makes it a real power-house!

Growing up

Do we ever really grow up? I’m not sure any more. I look at others and see some that have adopted this role, this image of how an adult should be like — but its more often than not tied into the whole A4 family thing or some superficial work profile. And since most Amiga fanatics are in their 40’s and 50’s (same age as Delphi hooligans, Turbo was released in 1983 same year as the Amiga came out), I guess this is when kids have grown up enough for people to go “wait a minute, what .. where is my Amiga!“.

But good things come to those who wait. If someone told me that I would one day work side by side with giants like David John Pleasance, Francois Lionet and the crew at FriendUp systems – I would never have believed them. A member of quartex in meetings with the head of Commodore? My teenage self would never have believed it. Both of these men, including all the tech guys at Commodore, Mark Sibly the guy behind BlitzBasic — these were my teenage heroes. And now I get to work with two of them. That is priceless.

As for growing up – if that means losing that spark, that trigger that when lost would render us incapable of enjoying things like the Amiga, reduced to a suit in a grey world of PCs – you know, then I’m happy to be exactly where I am. If you can go to work wearing an Amiga T-Shirt, tracker music on your iPod, a family you love at home, cool people to work with – I would call that a wrap.

And looking at the hundreds and thousands of people returning to the Amiga after 30 years in the desert – something tells me I wont be alone .. 😉

 

The Smart Desktop

March 5, 2017 Leave a comment

Right, if you checked out my previous post you know that I have been working on a cloud desktop for a while. It’s basiclly an example that will ship with the next release of Smart Mobile Studio, but it’s also a labour of love since I’m a huge Amiga fan.

My good friend Gunnar Kristjansson, the author behind Amibian, the Linux distro that more or less turns your Raspberry PI3b into a real live Amiga, booting straight into UAE4All on ARM based devices – has been helping out with testing and general opinions as well. We have decided to fork the base and call that “Amibian.js” since it is capable of running real-life 68k Amiga software right in the browser (!)

screenshot

Anyways if you find this interesting head over to github and pull the source from my smart repository. You will need the NG version of the RTL to compile, but that should be in the Alpha channel next week (unless something needs more love before its released).

Why should this matter?

The cloud desktop is not trying to be something it’s not. It’s meant as a conceptual example, meaning that it demonstrates what can be achieved (and expanded upon) with Smart Mobile Studio.

Having said that, it’s not just a pretty face! Making sexy css graphics is easy these days, but our RTL adds a bit of depth to the whole paradigm.

First of all, I implemented a virtual filesystem on top of Localstorage. This means that files can be read/written to the browser cache — and it will all be stored inside a single serialized item. This would be what people call “ram disk”. Essentially the capacity to store files in ram.

screenshot2

The same filesystem, which is naturally based on a common ancestor class, talks to the node.js backend server. Which means that once the server is finished – the desktop will have access to a region of the actual harddisk. This is where the fun starts!

What about applications

The desktop shows 3 windows when you start it. The first is a fully working text-editor. The second is a Deluxe Paint clone (pixel based editor) and last but not least – a music tracker (player) application.

Now all of these are pure JavaScript. Some of them are hosted elsewhere. So this is just an example of how easy it is to bind together local and external services.

Lets pixel like its 1999!

Lets pixel like its 1999!

I will naturally, when time allows, isolate these locally as a part of the repository – so that the applications can start as normal. They need to be adapted, because right now they run inside their own IFrame – which cause problems with window focus (as you will notice).

What can I use it for?

It should be a fairly good starting point if you want to create your own windowing cloud applications. I have given you the basics, like windowing classes, preferences and such things. The code is easy to read and play with – so it should be a good place to begin.

Personally I use this in my NAS developments. I discovered that my Asustor NAS (which has a similar concept) is a bit shit – so I decided I could write a better one myself.

Thankfully Gunnar Kristjannson, the guy behind Amibian, is helping me out with the Linux distro – and it boots directly into this system (if you have a screen attached). You can also connect to the NAS via ordinary http. And from that interface I can now control what the NAS does. And its all thanks to node.js and SMS.

The cool part? Replace the NAS with your favorite cloud supplier and you get the idea 😉

 

Smart Pascal: A real life desktop

February 16, 2017 6 comments

Every now and then I get feedback like “can SMS use jQuery?” or “Why don’t you use Sencha’s widgets“. And it just continues with everyone having their own favorite framework that they want SMS to adopt.

First of all, this is to completely misunderstand the architecture of Smart. You can use any framework you like. But you also have to sit down and write some code to incorporate it into the VJL. And believe me it’s not that hard.

The workbench

One of the cooler demo’s that I have been working on for .. oh, 8 hours now, is more or less a full Amiga OS 4 desktop clone. Sounds useless right? Well not really. It demonstrates some fundamental concepts:

  • How to create windows just like eh, windows have
  • How to display file-items in a listview
  • How to create controls outside the form so they remain unaffected by form switching
  • How to host demos and pre-compiled applications inside forms

But why stop there? Why not setup a node.js server and have the desktop act as a front-end to your server?

Always fun to see how far we can push the system

Check out this video on youtube 🙂

NAS front end

If you go out and buy a NAS today, chances are it comes with an HTML interface. So once you plug it into your router, you can browse to it and control the device via the browser.

Ring a bell? That is exactly what I’m doing right here. And it took me less than one working day to get this up and running. Here is how it looks right now:

Running a full remake of a classic demo in a window. No problem. You can run as much code as memory can hold

Running a full remake of a classic demo in a window. No problem. You can run as much code as memory can hold

But I want widget set x, y, z!

Had I done you the disservice of using Sencha or jQueryUI or whatever widget framework, you would have been stuck with that forever. Instead, you get a VCL like framework that is build in a way that ensures – that it can absorb and integrate any UI.

But let’s get back to that desktop.

You probably think: Ok so you have a fake desktop in a browser, it can run some JS demos and look cool. But so what?

You don’t get it. Did you know that X, the display system on Linux is by client / server system by default? Did you know that the entire Linux desktop is just an X client that connects to the server (the server being in the same distro) in order to do it’s business? If you start looking at what you can do – as opposed to what you imagine is impossible, there is a lot of cool stuff you can do for your company right now.

Doing some GPU profiling and watching the callstack

Doing some GPU profiling and watching the callstack

The first thing you need is to set up a server. A node.js server of course, that the website can talk to. Heck you can even do OAuth2 calls to dropbox and whatever online service you like – and get that on your HTML5 desktop without node.

Node however allows you to get system-level access. Listing files, loading files, saving files and even loading programs. Programs here being compiles Smart applications that the desktop can inject and execute inside a “window”.

Whenever you need something executing on the server – well then you call the server via websocket.

If node.js doesn’t do it for you then write it in Delphi or C++, it doesnt really matter. What matters is that you have a universal access point in the browser.

A quick visit to npm and git and you can download fully functional text-processors and large-scale HTML5 applications that does exactly what OpenOffice does. And with that websocket connection to your back-end, you have a real-life solution on your hands.

Here is a quick and dirty storage device API I made. Ram-disk will just store data to a B-Tree based “fake” in-memory filesystem. The real deal will come via websocket on the server.

unit Wb.desktop.Devices;

interface

uses
  System.Types, System.Types.Convert,
  System.Streams, System.Reader, System.Writer,
  System.Stream.Reader, System.Stream.Writer,
  System.Time,
  SmartCL.System,
  SmartCL.Time;

type

  TWbStorageDevice        = class;
  TWbStorageDeviceRamDisk = class;
  TWbDeviceManager        = class;
  TWbCustomFileSystem     = class;
  TWbVirtualFileSystem    = class;
  TWbRemoteFileSystem     = class;
  TWbLocalFileSystem      = class;
  TWbStorageDeviceClass = class of TWbStorageDevice;

  TWbCustomFileSystem = class(TObject)
  end;

  TWbVirtualFileSystem = class(TWbCustomFileSystem)
  end;

  TWbRemoteFileSystem = class(TWbCustomFileSystem)
  end;

  TWbLocalFileSystem = class(TWbCustomFileSystem)
  end;

  /* Requirements for using a device */
  TWbStorageDeviceOptions = set of
    (
      doRequireLogin, // Require authentication before Mount()
      doReadOnly      // Device is read-only
    );

  /* Filesystem access rights */
  TWbStorageDeviceAccess  = set of
    (
      daNone,       // none
      daReadOnly,   // read files only
      daReadWrite,  // read and write [create]
      daExecute     // can execute
    );

  TWbAuthenticatedEvent = procedure (Sender: TWbStorageDevice; Access: TWbStorageDeviceAccess);
  TWbMountEvent = procedure (Sender: TWbStorageDevice);

  /* Abstract storage device */
  TWbStorageDevice = class(TObject)
  private
    FId:            string;
    FName:          string;
    FFileSystem:    TWbCustomFileSystem;
    FOptions:       TWbStorageDeviceOptions;
    FMounted:       boolean;
    FAuthenticated: boolean;
    FManager:       TWbDeviceManager;
  protected
    procedure   SetName(const NewName: string); virtual;
    procedure   SetIdentifier(const NewId: string); virtual;
    procedure   SetFileSystem(const NewFileSystem: TWbCustomFileSystem); virtual;
    function    GetFileSystem: TWbCustomFileSystem; virtual;
    procedure   SetOptions(const NewOptions: TWbStorageDeviceOptions); virtual;
    function    GetOptions: TWbStorageDeviceOptions; virtual;
    procedure   SetAuthenticated(const NewState: boolean); virtual;
  public
    property    Name: string read FName;
    property    Identifier: string read FId;
    property    FileSystem: TWbCustomFileSystem read GetFileSystem;
    property    Options: TWbStorageDeviceOptions read GetOptions;
    property    Mounted: boolean read FMounted;
    property    Authenticated: boolean read FAuthenticated;
    property    DeviceManager: TWbDeviceManager read FManager;

    procedure   Authenticate(UserName, Password: string; const Success: TWbAuthenticatedEvent); overload;
    procedure   Authenticate(UserName, Password, Domain: string; Success: TWbAuthenticatedEvent); overload;
    procedure   Authenticate(AuthKey: string; Success: TWbAuthenticatedEvent); overload;

    procedure   Mount(const Success: TWbMountEvent);
    procedure   UnMount;

    constructor Create(const Manager: TWbDeviceManager); virtual;
    destructor  Destroy; override;
  end;

  /* RAM DISK */
  TWbStorageDeviceRamDisk = class(TWbStorageDevice)
  protected
    function    GetFileSystem: TWbCustomFileSystem; override;
  public
    constructor Create(const Manager: TWbDeviceManager); override;
  end;

  /* Cache disk */
  TWbStorageDeviceCache = class(TWbStorageDevice)
  protected
    function    GetFileSystem: TWbCustomFileSystem; override;
  public
    constructor Create(const Manager: TWbDeviceManager); override;
  end;

  TWbDeviceManager = class(TObject)
  private
    FClasses: array of TWbStorageDeviceClass;
    FObjects: array of TWbStorageDevice;
  public
    procedure RegisterDevice(const DeviceClass: TWbStorageDeviceClass);

    property  Count: integer read ( FObjects.Count );
    property  Device[const Index: integer]: TWbStorageDevice read ( FObjects[Index] ); default;

    destructor Destroy; override;
  end;

implementation

//#############################################################################
// TWbDeviceManager
//#############################################################################

destructor TWbDeviceManager.Destroy;
begin
  while FObjects.Count >0 do
  begin
    FObjects[0].free;
    FObjects.Delete(0,1);
  end;
  FClasses.Clear();
  inherited;
end;

procedure TWbDeviceManager.RegisterDevice(const DeviceClass: TWbStorageDeviceClass);
begin
  if FClasses.IndexOf(DeviceClass) < 0 then
  begin
    FClasses.add(DeviceClass);
    FObjects.add( DeviceClass.Create(self) );
  end;
end;

//#############################################################################
// TWbStorageDeviceCache
//#############################################################################

constructor TWbStorageDeviceCache.Create(const Manager: TWbDeviceManager);
begin
  inherited Create(Manager);
  SetName('DH0');
  SetIdentifier('{2D58F4D9-D8FE-434C-AC32-8B27EEC0AEE2}');
  SetOptions([doReadOnly]);
end;

function TWbStorageDeviceCache.GetFileSystem: TWbCustomFileSystem;
begin
  result := inherited GetFileSystem();
  if result = nil then
  begin
    result := TWbVirtualFileSystem.Create;
    SetFileSystem(result);
  end;
end;

//#############################################################################
// TWbRamDisk
//#############################################################################

constructor TWbStorageDeviceRamDisk.Create(const Manager: TWbDeviceManager);
begin
  inherited Create(Manager);
  SetName('Ram-Disk');
  SetIdentifier('{2E6D58D0-A0C3-4D62-8AC4-0300619418A6}');
  SetOptions([]);
end;

function TWbStorageDeviceRamDisk.GetFileSystem: TWbCustomFileSystem;
begin
  result := inherited GetFileSystem();
  if result = nil then
  begin
    result := TWbVirtualFileSystem.Create;
    SetFileSystem(result);
  end;
end;

//#############################################################################
// TWbStorageDevice
//#############################################################################

constructor TWbStorageDevice.Create(const Manager: TWbDeviceManager);
begin
  inherited Create;
  FManager := Manager;
end;

destructor TWbStorageDevice.Destroy;
begin
  if FFileSystem <> nil then
    FFileSystem.free;
  inherited;
end;

procedure TWbStorageDevice.Mount(const Success: TWbMountEvent);
begin
  if FMounted then
    UnMount;

  FMounted := true;

  if assigned(Success) then
  begin
    TW3Dispatch.Execute( procedure ()
      begin
        Success(self);
      end, 100);
  end;
end;

procedure TWbStorageDevice.UnMount;
begin
  if FMounted then
  begin
    FMounted := false;
  end;
end;

procedure TWbStorageDevice.SetAuthenticated(const NewState: boolean);
begin
  FAuthenticated := NewState;
end;

procedure TWbStorageDevice.SetOptions(const NewOptions: TWbStorageDeviceOptions);
begin
  FOptions := NewOptions;
end;

function TWbStorageDevice.GetOptions: TWbStorageDeviceOptions;
begin
  result := FOptions;
end;

procedure TWbStorageDevice.SetName(const NewName: string);
begin
  FName := NewName;
end;

procedure TWbStorageDevice.SetIdentifier(const NewId: string);
begin
  FId := NewId;
end;

procedure TWbStorageDevice.SetFileSystem(const NewFileSystem: TWbCustomFileSystem);
begin
  FFileSystem := NewFileSystem;
end;

function TWbStorageDevice.GetFileSystem: TWbCustomFileSystem;
begin
  result := FFileSystem;
end;

procedure TWbStorageDevice.Authenticate(UserName, Password: string; const Success: TWbAuthenticatedEvent);
begin
end;

procedure TWbStorageDevice.Authenticate(UserName, Password, Domain: string; Success: TWbAuthenticatedEvent);
begin
end;

procedure TWbStorageDevice.Authenticate(AuthKey: string; Success: TWbAuthenticatedEvent);
begin
end;

end.

So. Writing the foundation of a NAS front-end in Smart, a virtual desktop with a windowing toolkit took me less than 8 hours. How long would it take you in vanilla JS?

So forgive me if I dont take jQuery serious.

Smart Pascal: Changes

February 6, 2017 3 comments

The changes to Smart Mobile Studio over the past 12 months have been tremendous. At first glance they might not seem huge, but if you do a raw compare on the upcoming RTL and the now more than a year old RTL – I think you will find that there are very few places in the code where I havent done improvements.

In this post I will try to collect some of the changes. A final change log will be released together with the update, but at least this “preview” will tell you something about the hours, days, weeks and months I have put into the RTL.

The Smart Lab, this is where most of my ideas turn into code :)

The Smart Lab, this is where most of my ideas turn into code 🙂

There will also be changes to the IDE; We have 3 members that both have added new features in the past, and members that are adding changes right now. The immediate being an update from Chromium Embedded CEF2 to CEF4, which is a huge change in speed and preview quality. And there are other more important changes being worked on, the most pressing being the designer and “live” rendering of controls.

Ok, let’s just dive into it!

RTL Changes

  • The units that make up the RTL are now organized according to a namespace scheme
    • Visual units are prefixed with SmartCL
    • Universal units are prefixed with System
    • Low level API units for node.js are prefixed with NodeJS
    • High level Node classes are prefixed with SmartNJ
  • Better fragmentation: Code that was previously a part of one very large unit has been divided into smaller files to avoid large binaries. For example:
    •  TRect, TPoint etc. are now in System.Types.Graphics
    • System.Time is extended with SmartCL.Time which gives graphical timing functions like TW3Dispatch.RequestAnimationFrame()
  • The unit System.Objects has been added, isolating classes that work both in browsers, node and headless runtime environments:
    • TW3ErrorObjectOptions
    • TW3ErrorObject
    • TW3OwnedErrorObject
    • TW3HandleBasedObject
  • TW3Label has been completely re-written and is now faster, has synchronized state management, resize on ready-state and much more
  • The unit System.NameValuePairs was added. This implements a traditional name/value list (or dictionary) that is used by various classes and standards throughout the RTL (http headers being an example)
  • A show stopping bug in SmartCL.Effects has been fixed. Effects now works as expected (simply add SmartCL.Effects to your uses clause)
  • TFileStream has been added to SmartNJ.Streams. This is unique for node.js since only node (and some hybrid runtime engines) support direct file access.
  • TBinaryData is extended in SmartNJ.Streams.pas to emit and consume node.js buffers which are different from traditional JS buffers. ToNodeBuffer() and FromNodeBuffer() allows you to consume special node buffer types.
  • A full software tweening engine has been written from scratch in pure Smart Pascal and added to the SmartCL namespace. The following units have been added:
    • SmartCL.Tween
      • TW3TweenElement
      • TW3TweenEngine
      • function TweenEngine: TW3TweenEngine
    • SmartCL.Tween.Effect
      • TW3CustomEffect
      • TW3CustomTweenEffect
      • TW3ControlTweenEffect
      • TW3MoveXEffect
      • TW3MoveYEffect
      • TW3MoveToEffect
      • TW3ColorMorphEffect
      • TW3CustomOpacityEffect
      • TW3FadeInEffect
      • TW3FadeOutEffect
      • TW3OpacityEffect
    • SmartCL.Tween.Ease
      • TW3TweenEase
      • TW3TweenEaseLinear
      • TW3TweenEaseQuadIn
      • TW3TweenEaseQuadOut
      • TW3TweenEaseQuadInOut
      • TW3TweenEaseCubeIn
      • TW3TweenEaseCubeOut
      • TW3TweenEaseCubeInOut
      • TW3TweenEaseQuartIn
      • TW3TweenEaseQuartOut
      • TW3TweenEaseQuartInOut
      • TW3TweenEaseQuintIn
      • TW3TweenEaseQuintOut
      • TW3TweenEaseQuintInOut
      • TW3TweenEaseSineIn
      • TW3TweenEaseSineOut
      • TW3TweenEaseSineInOut
      • TW3TweenEaseExpoIn
      • TW3TweenEaseExpoOut
      • TW3TweenEaseExpoInOut
      • TW3TweenEaseCollection
      • function TweenEaseCollection: TW3TweenEaseCollection
  • Support for require.js (simplified resource management and more) is now a part of the RTL. The unit SmartCL.Require gives you:
    • TRequireError
    • TW3RequireJSConfig
    • TW3RequireJS
    • function Require: TW3RequireJS;
    • procedure Require(Files: TStrArray);
    • procedure Require(Files: TStrArray; const Success: TProcedureRef);
    • procedure Require(Files: TStrArray; const Success: TProcedureRef; const Failure: TW3RequireErrHandler);
  • SmartCL (browser) based communication has been consolidated into more appropriate named units:
    • SmartCL.Net.http
    • SmartCL.Net.http.headers
    • SmartCL.Net.websocket
    • SmartCL.Net.jsonp
    • SmartCL.Net.socketIO
    • SmartCL.Net.rest
  • A callback bug in the REST api has been fixed (called wrong handler on exit)
  • Suppport for mutation events has been added to the RTL. This can be found in the unit SmartCL.Observer. Mutation observer allows you to listen for changes to any HTML element, its properties or attributes. This is a very important unit with regards to data-aware controls:
    • TMutationObserverOptions
    • TMutationObserver
  • A fallback shim for mutation observing has been added to the $RTL\Shims folder. This ensures that mutation events can be observed even on older browsers.
  • A MSIE legacy shim that patches all missing IE functionality from the current back to IE5 has been added. This allows Smart applications to execute without problems on older browsers (which there are surpricingly many of)
  • The unit SmartCL.Styles has been added. This contains classes and methods that create a stylesheet at runtime. The stylesheet is deleted when the object instance is disposed. This allows for easier styling from within your code rather than having to pre-define it in the global css file.
  • Polygon helper classes have been isolated in SmartCL.Polygons. These expose functionality to TPointArray and TPointFArray:
    • TPolygonHelper
    • TPolygonFHelper
  • Support for socket.io has been added for the browser. The unit SmartCL.Net.SocketIO contains the following client class and methods:
    • TW3SocketIOClient
      • procedure Connect(RemoteHost: string)
      • procedure Disconnect
      • procedure Emit(EventName: string; Data: variant)
      • procedure On(EventName: string; Handler: TW3SocketIOHandler)
  • Support for socket.io has been added to the new SmartNJ namespace as well, allowing you to write fast, good performance SocketIO servers.
  • Support for WebSocketIO, an amalgamation of classic WebSocket and SocketIO has been added. This is the default socketio server type for our node.js system
  • Support for user-attributes has been added and isolated in SmartCL.Attributes. This allows you to easily add, read, write and remove custom attributes to any tag. This is a very important feature that is used both by the effects framework – and also by the database framework.
  • SmartCl.Buffers unit has been updated. Now uses the latest methods of the RTL and now works on all browsers, not just webkit and Firefox.
  • $RTL\SmartCL\Controllers\SmartCL.Scroll.Momentum has been removed
  • $RTL\SmartCL\Controllers\SmartCL.Scroll.plain has been removed
  • All visual controls have been updated, rewritten and thoroughly tested
  • The RTL now has support for ACE, the #1 JavaScript code editor. This is more or less a JavaScript implementation of Delphi’s Synedit, but with functionality closer to Sublime. Ace comprises over 100 files. In this first release we have focused on getting a respectable basis in place.
    • The wrapper units is SmartCL.AceEditor, which gives you the following classes:
      • TW3AceMode
      • TW3AceTheme
      • TW3AceEditor
        • Text: string
        • SelectedText: string
        • LineCount: integer
        • WordWrap: boolean {get; set;}
        • ShowPrintMargin: boolean {get; set;}
        • EditorMode: TW3AceMode {get; set;}
        • Theme: TW3AceTheme {get; set;}
    • Modes and syntax we added support for:
      • AceEdit.Mode.Abap
      • AceEdit.Mode.Abc
      • AceEdit.Mode.ActionScript
      • AceEdit.Mode.Ada
      • AceEdit.Mode.Apache
      • AceEdit.Mode.AppleScript
      • AceEdit.Mode.Ascii
      • AceEdit.Mode.AutoHotKey
      • AceEdit.Mode.Base
      • AceEdit.Mode.BatchFile
      • AceEdit.Mode.C9Search
      • AceEdit.Mode.Cirru
      • AceEdit.Mode.Clojure
      • AceEdit.Mode.Cobol
      • AceEdit.Mode.Coffee
      • AceEdit.Mode.ColdFusion
      • AceEdit.Mode.Cpp
      • AceEdit.Mode.CSharp
      • AceEdit.Mode.CSS
      • AceEdit.Mode.Curly
      • AceEdit.Mode.D
      • AceEdit.Mode.Dart
      • AceEdit.Mode.Diff
      • AceEdit.Mode.django
      • AceEdit.Mode.Pascal
      • AceEdit.Mode.VbScript
      • AceEdit.Mode.X86Asm
    • Themes we support:
      • AceEdit.Theme.Ambiance
      • AceEdit.Theme.Base
      • AceEdit.Theme.Chaos
      • AceEdit.Theme.Chrome
      • AceEdit.Theme.CloudMidnight
      • AceEdit.Theme.Clouds
      • AceEdit.Theme.Cobalt
      • AceEdit.Theme.CrimsonEdit
      • AceEdit.Theme.Dawn
      • AceEdit.Theme.Dreamweaver
      • AceEdit.Theme.eclipse
      • AceEdit.Theme.Github
      • AceEdit.Theme.Monokai
      • AceEdit.Theme.Twilight
  • The RTL has a completely re-written ready-state engine. Since JavaScript is ASync by nature, the handle or reference to the TAG a class manages may not be ready during the constructor. The ready-state engine waits in the background for the handle to become valid and the control available in the DOM (document object model). It then sets the appropriate component-state flags and issues a resize to ensure that the control looks as it should.
  • Support for Control-State (which is common in Delphi and Lazarus) has been added to visual controls. The following state flags can be read or set:
    • csCreating
    • csLoading
    • csReady
    • csSized
    • csMoved
    • csDestroying
  • TW3CustomControl now supports creation-flags. This is also a feature common to Delphi’s VCL and Lazarus’s LCL. It allows you to set some flags that disable fundamental behavior when you don’t need it. For instance, a label which will always have a fixed size or be in a fixed place does not need Resize() management. Turning off resize-checking makes you application execute faster. The following creation flags can be defined:
    • cfIgnoreReadyState
    • cfSupportAdjustment
    • cfReportChildAddition
    • cfReportChildRemoval
    • cfReportMovement
    • cfReportResize
    • cfAllowSelection
    • cfKeyCapture
  • To enable or disable component states, use the following methods of TW3CustomControl:
    • procedure AddToComponentState(const Flags: TComponentState)
    • procedure RemoveFromComponentState(Const Flags: TComponentState)
  • Support for keyboard input and charcode capture has been added to the RTL. Simply add the flag cfKeyCapture to the CreationFlags() function of your custom-control and the control’s OnKeyPress event will fire when the control has focus and a key is pressed. By default this is turned off.
  • Support for text-selection and disabling text-selection has been added. Simply add or remove cfAllowSelection from your controls CreationFlags() function. By default text selection is turned off (except for obvious controls like text-edit).
  • TW3CustomControl now has a zIndex property. This represents the z-order of a control (which control is in front of others).
  • The function TW3CustomControl.GetZOrderList(sort: boolean) can be called to get a sorted or un-sorted list of child elements. This provides both the handle for each control and its zindex value.
  • The method TW3CustomControl.Showing() has been updated, it now takes height for IFrame and CSS4 GPU positioning (read: it will understand that its offscreen even if you use CSS3 to move it).
  • TW3CustomControl now has a Cursor property. This allows you to read and set the mouse cursor for a control. The following cursors are supported:
    • crAuto
    • crDefault
    • crInherited
    • crURL
    • crCrossHair
    • crHelp
    • crMove
    • crPointer
    • crProgress
    • crText
    • crWait
    • crNResize
    • crSResize
    • crEResize
    • crWResize
    • crNEResize
    • crNWResize
    • crNSResize
    • crSEResize
    • crSWResize
    • crEWResize
  • To simplify mouse cursor handling, the class TW3MouseCursor, which is a static class, has been added to SmartCL.System.pas. It expose the following methods:
    • function  CursorByName(const CursorName: string): TCursor
    • function  NameByCursor(const Cursor: TCursor): String
    • function  GetCursorFromElement(const Handle: TControlHandle): TCursor
    • procedure SetCursorForElement (const Handle: TControlHandle; const Cursor: TCursor)
  • TW3CustomControl have two new methods to deal with cursor changes. These are virtual and can be overriden:
    •  function GetMouseCursor: TCursor;
    • procedure SetMouseCursor(const NewCursor: TCursor);
  • Basic device capabillity examination support has been added. The class TW3DOMDeviceCapabilities has been added to SmartCL.System. This exposes the following methods and properties:
    • DevicePixelRatio: float
    • DisplayPixelsPerInch: TPixelsPerInch
    • GetMouseSupport: boolean
    • GetTouchSupport: boolean
    • GetGamePadSupport: boolean
    • GetKeyboardSupported: boolean
    • GetDevicePixelRatio: float
    • GetDisplayPixelsPerInch: TPixelsPerInch
  • In order to make the VJL more architectually compatible with Delphi’s VCL and Freepascal’s LCL – TW3Component as name has been pushed back. TW3TagObj used to be the root class for visual components, followed by TW3Component, TW3MovableControl and finally TW3CustomControl. However, since we want to use TW3Component as a common non-visual control, this name was pushed back. So TW3TagObj now inherits from TW3Component, and TW3TagContainer has taken the place TW3Component once had. Here is the new inheritance chain
    • TW3CustomComponent
      • TW3Component
        • TW3TagObj
          • TW3TagContainer
            • TW3MovableControl
              • TW3GraphicControl
              • TW3CustomControl
  • TW3Component is now the basis for non-visual components, which opens up for a whole new set of controls that can be dragged onto a form or datamodule. Sadly Datamodules did not make it into this update, but it is the next logical step – hopefully we will have it in place with the IDE update that will appear after this.
    • Controllers will eventually be re-incarnated as TW3Component’s
    • Rouge classes will be implemented as TW3Components
    • Database classes will become non-visual TW3Components
    • Communication classes will become non-visual TW3Components
  • TW3Timer is moved to the unit System.Time.pas
  • TW3Timer now inherits from TW3Component
  • The following procedures have been deprecated and moved to System.Time.pas. Please use the corresponding methods in TW3Dispatch (Note: The w3_* methods will be deleted in the next update!):
    • w3_Callback = TW3Dispatch.Execute()
    • w3_SetInterval = TW3Dispatch.SetInterval()
    • w3_ClearInterval = TW3Dispatch.ClearInterval()
    • w3_SetTimeout = TW3Dispatch.SetTimeOut()
    • w3_ClearTimeout = TW3Dispatch.ClearTimeOut()
  • TW3Dispatch, which is a class dealing with timing and scheduling, now support the following new methods:
    • class function JsNow: JDate;
    • class function Ticks: integer;
    •  class function TicksOf(const Present: TDateTime): integer;
    • class function TicksBetween(const Past, Future: TDateTime): integer;
    • class procedure RepeatExecute(const Entrypoint: TProcedureRef; const RepeatCount: integer; const IntervalInMs: integer);
  • SQLite has been recompiled from C to JS and is now version 3.8.7.4
  • The SQLite units have been moved from SmartCL.SQLite to System.SQLite
  • The following SQLite classes have been given an overhaul:
    • TSQLiteDatabase
    • TSQLiteDBObject
    • TSQLiteRowValues
    • TSQLiteResult
    • TSQLParamData
    • TSQLitePair
    • TSQLiteParams
    • TSQLiteStatement
  • The SQLiteInitialize() global method is no longer requires and has been removed. When you include System.SQLite.pas in your project, the whole database engine is automatically linked and loaded into memory on execute. Use the SQLiteReady() global function to check if the database engine has loaded before use.
  • TReader, TWriter which was exclusively a re-implementation based on LCL and Delphi has been fragmented. TReader and TWriter are now base classes that takes an interface as parameter in their constructor (IBinaryTransport). Both TStream and TBinaryData, which are the easiest ways of dealing with binary files and memory – implements the IBinaryTransport interface.
  • TStreamReader and TStreamWriter has been added to the RTL, these are implemented in System.Stream.Reader.pas and System.Stream.Writer.pas. When working with streams, make sure you use TStreamReader and TStreamWriter. Using TReader and TWriter directly may cause problems if you are not intimately familiar with the RTL.
  • The unit System.Structure has been added to the RTL. This is to simplify working with structured records, abstracting you from the underlying format. The following classes are exposed in System.Structure:
    • EW3Structure [exception]
    • TW3Structure
      • procedure WriteString(Name: string; Value: string; const Encode: boolean);
      • procedure WriteInt(const Name: string; value: integer);
      • procedure WriteBool(const Name: string; value: boolean);
      • procedure WriteFloat(const Name: string; value: float);
      • procedure WriteDateTime(const Name: string; value: TDateTime);
      • function ReadString(const Name: string): string;
      • function ReadInt(const Name: string): integer;
      • function ReadBool(const Name: string): boolean;
      • function ReadFloat(const Name: string): float;
      • function ReadDateTime(const Name: string): TDateTime;
      • function Read(const Name: string): variant;
      • procedure Write(const Name: string; const Value: variant);
      • procedure Clear;
      • procedure SaveToStream(const Stream: TStream);
      • procedure LoadFromStream(const Stream: TStream);
      • procedure LoadFromFile(Url: string; const callback: TW3StructureLoadedCallback);
  • The unit System.Structure.JSON implements a TW3Structure class that stores data as JSON. The data is stored in memory as a single record, but naturally you can add further records as child properties – and then export the complete data as binary format to a stream, or as a single string through the ToString() method.
  • The unit System.Structure.XML implements a TW3Structure class which emits XML. Since the XML interface in browsers is both unstable and wildly different between vendors, we use our own BTree engine to manage the data. The class emits valid XML and will also deal with sub-records just as TW3JSonStructure.
  • The unit System.JSON has been added, which contains the following classes and methods. TJSON is a highly effective wrapper over the built-in JSON API, making it easier to work with JSON in general. The constructor is highly overloaded so you can wrap existing JS elements directly. This class is used by TW3JSONStructure to simplify its work:
    • TJSONObjectOptions
    • EJSONObject [exception]
    • TJSONObject
    • TJSON
  • The Delphi parser library TextCraft has been added to the RTL. When I write added that is not really correct. TextCraft was first written in Smart and then post converted to Delphi. But the RTL now has the latest update of this library, making advanced, recursive text-parsing possible:
    • System.Text.Parser.pas
    • System.Text.Parser.Words.pas
  • TW3Borders has been re-coded and optimized.
    • TW3Border.EdgeString() now use a lookup table for maximum efficiency
    • TW3Border’s edges now use a lookup table to quickly map types to strings
    • All superfluous reference testing has been removed, resulting in much faster and efficient code
    • Styles are no longer read via the older w3_read/write mechanisms but rather directly from the controlhandle
  • Two new classes has been added to deal with border-radius.
    • All controls can set the radius for all edges using the older TW3MovableControl.BorderRadius property, but this has now been expanded on. The following classes has been added. TW3BorderRadius can be accessed via the new TW3CustomControl.EdgeRadius property, which creates an instance on demand (first access):
      • TW3BorderEdgeRadius
      • TW3BorderEdgeTopRadius
      • TW3BorderEdgeBottomRadius
      • TW3BorderRadius
  • TStringBuilder has been added to System.Type. It deviates some from the Lazarus and Delphi variations because it adds functions that actually makes sense ! 🙂
  • All custom controls have a cool background class that allows you to set color, add images and export pixmaps. A new class called TW3ControlBackgroundSize has been added to this (TW3CustomControl.Background.Size) making things even easier. Especially detecting if a background is a graphic and its true size (!)
    • function  BackgroundIsImage: boolean;
    • property Mode: TW3ControlBackgroundSizeMode
    • property Width: integer;
    • property Height: integer;
  • TW3Constraints (inherits from TW3OwnedObject) has been added to the RTL. This is a class that deals with size constraints for controls. Please note that this is experimental (!) When active it forces size restrictions much like the VCL and LCL — but it can cause havoc in a design that auto-scales (!). It implements the following:
    • property Enabled: boolean
    • property MinWidth: integer
    • property MinHeight: integer
    • property MaxWidth: integer
    • property MaxHeight: integer
    • procedure   ApplyToOwner
    • function GetMaxWidth: integer;
    • function GetMaxHeight: integer;
    • procedure SetMaxWidth(const NewMaxWidth: integer);
    • procedure SetMaxHeight(const NewMaxHeight: integer);
    • function GetMinWidth: integer;
    • function GetMinHeight: integer;
    • procedure SetMinWidth(aValue: integer);
    • procedure SetMinHeight(aValue: integer);
    • procedure   SetEnabled(const NewValue: boolean); virtual;
  • Event classes has finally been added! Need another OnClick event? just Create an event object and attach it to your control. The following event classes are now in SmartCL.Events :
    • TW3DOMEvent
    • TW3StandardDOMEvent
    • TW3MouseEnterEvent
    • TW3MouseLeaveEvent
    • TW3MouseDownEvent
    • TW3MouseMoveEvent
    • TW3MouseUpEvent
    • TW3ElementRemovedEvent
    • TW3ElementAddedEvent
    • TW3ElementContextMenuEvent
    • TW3ElementClickEvent
    • TW3ElementDblClickEvent
    • TW3ElementMouseOverEvent
    • TW3ElementKeyDownEvent
    • TW3ElementKeyPressEvent
    • TW3ElementKeyUpEvent
    • TW3ElementChangeEvent
    • TW3ElementMouseWheelEvent
    • TW3DOMEventAPI
      • class procedure RegisterEvent(Handle: TControlHandle; EventName: string; EventHandler:TW3JSEventHandler; Mode: TW3DOMEventMode); static;
      • class procedure UnRegisterEvent(Handle: TControlHandle; EventName: string; EventHandler:TW3JSEventHandler; Mode: TW3DOMEventMode); static;
  • Since events are not DOM or browser bound, they also occur in node.js and other JavaScript virtual machines (and you can also register your own events just like you do in Delphi or C#). As such I added System.Events.pas which contains the basic functionality for OOP events. The following classes and methods are added:
    • TW3SystemEventObject
      • procedure Attach(NameOfEvent: string);
      • procedure Detach;
      • property  Attached: boolean
      • property  EventName: string
  • TW3CustomBrowserAPI.Styles references “window.styles” which is no longer supported by browsers. This has been updated to “window.document.head.style”
  • BrowserAPI() now expose key browser objects as actual class objects. The following objects are exposed as handles only (as they have been for some time):
    • Document
    • Body
    • Window
    • Styles
    • Console
    • Navigator
    • Self
    • Event
  • Besides the above handles, we now expose direct access to the mapped class objects directly. The W3C units have been updated to reflect modern browsers (so the objects have the latest methods exposed):
    • DocumentObject: JDocument
    • BodyObject: JHTMLElement
    • WindowObject: JWindow
    • StylesObject: JCSSStyleDeclaration
    • NavigatorObject: JNavigator
    • EventObject: JEvent
  • Added missing W3C.localStorage unit file which expose the localstorage API. This can now be accessed via BrowserAPI().WindowObject.localStorage. It supports the following functions:
    • property  key[const keyId: string]: variant
    • property  length: integer
    • procedure setItem(const key: string; const Value: variant)
    • procedure removeItem(const key: string)
    • procedure clear
    • function  valueOf: variant [* prototype]
    • function  hasOwnProperty(const key: string): boolean [* prototype]
  • TW3Image now has a fitstyle property which wraps the “object-fit” css style. This gives you finer control over how the image is presented inside the control (or the background of a control). The following presentation styles are supported:
    •  fsNone
      Image will ignore the height and width of the parent and retain its original size.
    • fsFill
      This is the default value which stretches the image to fit the content box, regardless of its aspect-ratio.
    • fsContain
      Increases or decreases the size of the image to fill the box whilst preserving its aspect-ratio.
    • fsCover
      The image will fill the height and width of its box, once again maintaining its aspect ratio but often cropping the image in the process.
    • fsScaleDown
      The control will compare the difference between fsNone and fsContain in order to find the smallest concrete object size.
  • TW3Image now supports binary IO besides the common HTML src property. The following methods have been added to TW3Image:
    • function  ToBuffer: TBinaryData;
    • function  ToStream: TStream;
    • function  ToDataUrl: string;
    • function  ToImageData: TW3ImageData;
    • procedure LoadFromURL(aURL: String);
    • procedure LoadFromImageData(Const Data:TW3ImageData);
    • procedure LoadFromBinaryData(const Memory:TBinaryData);
    • procedure LoadFromStream(const Stream:TStream);
    • procedure Allocate(aWidth,aHeight:Integer);
  • TW3Image now expose 3 new properties that are very helpful. The PixelWidth and PixelHeight gives you the actual size of an image, regardless of scale or viewport proportions. This is handy when using the Allocate() method to create a pixel-buffer that matches an existing image:
    • property  PixelWidth: integer
    • property  PixelHeight: integer
    • property  Complete: boolean
  • The following procedures and functions have been removed. Faster to assign and deal with events directly on the handle – no need for the extra call:
    • procedure w3_bind
    • procedure w3_bind2
    • procedure w3_unbind
    • procedure w3_unbind2
  • TStreamHelper has been added to SmartCL.System. It adds data objectification. Generating an internal URL to access the data via a link. These functions wraps the createObjectURL API (https://developer.mozilla.org/en-US/docs/Web/API/URL/createObjectURL) The following methods are in the helper:
    • function  GetObjectURL: string;
    • procedure RevokeObjectURL(const ObjectUrl: string);
  • TAllocationHelper has been added to SmartCL.System. It does exactly the same as TStreamHelper but for the memory class TAllocation (system.memory.allocation.pas).
  • TBinaryData which is a very flexible and fast memory buffer class (system.memory.buffer.pas), inherits from TAllocation – as such TAllocationHelper also affects this class.
  • The class TW3URLObject, which is a static class, has been added to SmartCL.System.pas. It wraps the same API as TStreamHelper and TAllocationHelper, but operates independent of medium. This class also adds forced download methods – where you can start a “save as” of binary data (or any data) with a single call. The class interface contains the following:
    • class function  GetObjectURL(const Text, Encoding, ContentType, Charset: string): string;
    • class function  GetObjectURL(const Text: string): string;
    • class function  GetObjectURL(const Stream: TStream): string;
    • class function  GetObjectURL(const Data: TAllocation): string;
    • class procedure RevokeObjectURL(const ObjectUrl: string);
    • class procedure Download(const ObjectURL: string; Filename: string);
    • class procedure Download(const ObjectURL: string; Filename: string;
      const OnStarted: TProcedureRefId);
  • TW3CustomControl now have automatic ZIndex assignment. On creation a ZIndex value is automatically assigned to each control (auto-inc number). The following methods have been added for that value:
    • function  GetZIndex: integer;
    • procedure SetZIndex(const NewZIndex: integer);
  • To better deal with z-order and the creation sequence of child elements, the following methods have been added to TW3CustomControl. This makes it easier to control how child elements stack and overlap:
    • function  GetZOrderList(const Sort: boolean): TW3StackingOrderList;
    • function  GetChildrenSortedByYPos: TW3TagContainerArray;
    • function  GetChildrenSortedByXpos: TW3TagContainerArray;
  • TW3CustomControl’s Invalidate() method has previously done nothing. It has been present for TW3GraphicControl (which inherits from TW3CustomControl) as means of redrawing or refreshing graphics.
    Invalidate for non-graphic controls now calls resize() to update the layout. While coders should use BeginUpdate / EndUpdate, there are times when a call to Invalidate() is all you need.
  • Text-shadow, which is a very effective CSS effect for text, has been isolated in its own class (TW3TextShadow) and is now exposed as TW3CustomControl’s TextShadow property. The class is created on-demand and will not consume memory until used.
  • TW3TagStyle, a class that parses and allows you to add, check and remove CSS style-rules to your control has been updated. The previous parsing routine has been replaced by a much faster string.split() call – removing the need for the for/next loop used to extract styles previously. This class is exposed as TW3CustomControl’s “TagStyle” property. Like Text-Shadow it is created on demand.
    Note: This replaces the older “CssClasses” property which has now been deprecated.
  • The TW3AnimationFrame class has been moved from SmartCL.Components.pas to SmartCL.Animation.pas which seems more fitting.
  • TControlHandleHelper is a strict helper-class for TControlHandle, which is the handle type used for HTML elements in the DOM. Many of the rouge utility functions that were scattered around the RTL have now been isolated here. It now gives you the following functionality:
    • function Defined: boolean;
    • function UnDefined: boolean;
    • function Valid: boolean;
    • function Equals(const Source: TControlHandle): boolean;
    • function Ready: Boolean;
    • function GetChildById(TagId: string): TControlHandle;
    • function GetChildCount: integer;
    • function GetChildByIndex(const Index: integer): TControlHandle;
    • function GetChildOf(TagType: string; var Items: TControlHandleArray): boolean;
    • function GetChildren: TControlHandleArray;
    • function HandleParent: TControlHandle;
    • function HandleRoot: TControlHandle;
    • procedure ReadyExecute(OnReady: TProcedureRef);
    • procedure ReadyExecuteEx(const Tag: TObject; OnReady: TProcedureRefEx);
    • procedure ReadyExecuteAnimFrame(OnReady: TProcedureRef);
  • function w3_GetCursorCSSName, w3_GetCursorFromCSSName has been removed from SmartCL.System.pas – use the methods in the static class TW3MouseCursor instead
  • w3_DOMReady, w3_CSSPrefix and w3_CSSPrefixDef has been removed from SmartCL.System.pas – use the methods in BrowserAPI instead
  • The RTL now has its own stylesheet classes. The unit SmartCL.CSS.StyleSheet.pas and SmartCL.CSS.Classes represents the CSS management units for the system.

    TSuperstyle (see below) generates animation styles on-the-fly and returns the name for it. You can add this name to a control with Control.TagStyle.add().

    TCSS is a helper class for building complex styles via code. TSuperstyle uses this to create the animation effects. Note that these two classes are meant as examples (!) SmartCL.CSS.StyleSheet contains the following classes:

    • TW3StyleSheet
      • property    Handle: TControlHandle
      • property    StyleSheet: JCSSStyleSheet
      • property    StyleRules: JCSSRuleList
      • property    Count: integer
      • property    Items[const Index: integer]: string
      • class function  CreateStyleId: string;
      • class function  CreateStylesheetId: string;
      • class function  CreateStyleElement: TControlHandle;
      • class procedure DisposeStyleElement(const Handle: TControlHandle);
      • class function DefaultStylesheet: TW3StyleSheet;
      • function GetSheetObject: THandle;
      • function GetRuleObject: THandle;
      • function GetCount: integer;
      • function GetItem(const Index: integer): string;
    • TSuperStyle
      • class function EdgeRound(Size: integer): string; overload;
      • class function EdgeRound(TopLeftP, TopRightP, BottomLeftP, BottomRightP: integer):String; overload;
      • class function EdgeTopaz: string;
      • class function EdgeAngaro: string;
      • class function AnimGlow(GlowFrom, GlowTo: TColor): string;
      • class procedure AnimStart(Handle: TControlHandle; animName: String);
    • TCSS
      • function KeyFrames: TCSS;
      • function From: TCSS;
      • function &To: TCSS;
      • function Enter: TCSS;
      • function Leave: TCSS;
      • function PercentOf(Value: integer): TCSS;
      • function IntOf(Value: integer): TCSS;
      • function ColorOf(Value: TColor): TCSS;
      • function &inc(Value: string): TCSS;
      • function CRLF: TCSS;
      • function OpenParam: TCSS;
      • function CloseParam: TCSS;
      • function Background: TCSS;
      • function BoxShadow(Left, Top, Right, Bottom: integer): TCSS;overload;
      • function BoxShadow(Left, Top, Right, Bottom: integer; Color:TColor):TCSS;overload;
      • function LinearGradientV(aFrom, aTo: TColor): TCSS;
      • function LinearGradientH(aFrom, aTo: TColor): TCSS;
      • function LinearGradientTL(aFrom, aTo: TColor): TCSS;
      • function LinearGradientTR(aFrom, aTo: TColor): TCSS;
      • function BeginComplexGradient(Angle: integer): TCSS;
      • function EndComplexGradient: TCSS;
      • function ColorPercent(PercentOf: integer; Color: TColor): TCSS;
      • function LinearGradientAngle(aFrom, aTo: TColor; const Angle: float): TCSS;overload;
      • function LinearGradientAngle(Colors: Array of TColor; const Angle: float): TCSS; overload;
      • function AnimationName(Name: string): TCSS;
      • function AnimationDuration(Secs, MSecs: integer): TCSS;
      • function AnimationInfinite: TCSS;
      • class function Make: TCSS;
      • function Assign(Value: string): TCSS;

Controllers

A new concept has been added to the RTL, namely something called controllers. These are classes that attach to a control and either alter its behavior or exposes functionality that is not available by default.

SmartCL.Controller.EdgeSense

This unit exposes the class TW3EdgeSenseController, which can be attached to any visual control. Once attached the controller will fire events whenever the mouse-pointer or touch is close to the border of the control. This is typically used by controls that is expected to be resized, like a grid header column.

SmartCL.Controller.Swipe

This unit provides the following classes:

  • TW3SwipeControllerOptions
  • TW3SwipeRange
  • TW3SwipeController

The TW3SwipeController attaches itself to any visual control and will trigger an event if a swipe gesture is performed on the target control. Since gesures such as swipe is one of the most common gestures, it made sense to isolate this in particular.

This controller is presently used by one of the new components, TW3Win10Header, which is a html5 implementation of the Windows 10 mobile header. This is a good-looking control that allows you to swipe through categories, much like a page-control for Windows applications, but better suited for mobile designs.

SmartCL.Controller.TagAttributes

This controller implements access to custom data attributes for any visual control. It is essentially the functionality of TW3ElementAttributes (unit: SmartCL.Attributes= but here implemented as a controller.

The class used by the RTL for this behavior today is TW3ElementAttributes, but this will be deprecated in the future in favour of the controller.

SmartCL.Scroll.IScroll

iScroll is now the default scrolling mechanism for visual Smart Mobile Studio applications. The new TW3ScrollWindow and TW3CustomScrollList visual classes all derive their fantastic accurate scrolling and movement capabilities from iScroll – which by many is regarded as the defacto JavaScript scrolling library.

To make iScroll easier to use when writing custom controls that don’t inherit from TW3ScrollWindow or TW3CustomScrollList – you can use the controller instead. Simply create an instance, attach it to the parent container (which contains the elements you want to scroll) and you pretty much have instant iOS / Android scrolling.

Note: $Libraries\IScroll\iScroll.js has been updated to iScroll 5.2.0, which works fine on older Android devices (there were some problems with iScroll before). I have also updated our class implementation to include ALL options, events and functionality.

New Visual Controls

Indeed. While more or more controls is going to be added once the IDE update is in place, we at least have a few new controls in place that can liven up your applications.

First of all, iScroll is now (as explained) the default scroll mechanism in Smart Mobile Studio. When you combine that with the already impressive list of effects (smartcl.effects.pas) writing controls that gives visual feedback when you touch them, that scroll smoothly with momentum, bounce and snap to – is now almost ridiculously simple.

TW3DBGrid

First up is the long-awaited DB grid. This is a fast paced grid that can show thousands of elements without any significant speed penalty. This is because it creates the elements as they come into view – and it uses a very effective cache system to keep track of which elements (rows / cols) have been created, and which needs some attention.

grid

It also allows you to resize and move columns as you would expect from a native grid, so hopefully this first incarnation is just what you expected.

TW3CategoryHeader

Next we have the fancy Windows 10 mobile category header control. This is a touch driven, swipe responsive header that shows both a category and a small ingress text.

categories

This is an excellent control to create with Application.Display as a parent. That way it remains on top regardless of form – and the user can quickly navigate through swipes.

TW3SimpleLabel

We have also added a more lightweight label control. This is best suited for static text that don’t need horizontal alignment. It is a lot more resource friendly than TW3Label which is a composite custom control.

Ace editor control

Ace, the #1 Javascript code editor, akin to Delphi’s SynEdit or the popular Sublime editor, has likewise been added. This should make a lot of things easier, especially if you are creating front-end code for your cloud services (which you can now write in pristine node.js).

ace

TW3LayoutControl

TW3LayoutControl is another nice addition. This is essentially a scrollbox with classical rulers top and left. Perfect for image display programs, paint programs, layout designers and similar tasks. You can override and adapt this heavily and make it the basis for a calendar grid if you like.

We have also completely updated, re-written and quality checked every single control in the RTL. They are now more robust, they update and synchronize with their ready-state more or less like native controls do – and I have spent quite some time making sure they behave as they should.

Oh and yes, there are more controls (this post is getting long).

NodeJS high-level classes

Without a doubt the feature that has received most attention in this update are the Node.JS high level classes and units.

The previous NodeJS namespace and project type gave you more or less the bare-bones of node.js. This could be quite confusing for developers coming straight from Delphi or Lazarus. Especially if you don’t really want to dive to deep into the world of JavaScript – but enjoy some abstraction and ease of use.

The new SmartNJ namespace, where the files also are prefixed with the sane, implements common classes for both clients and servers. Writing an HTTP server is for instance very simple – and the code is in many cases as close to the other server types as possible.

The idea is that if you know how to write one type of server, you should also be able to write other types as well; because they share the same ancestor and commonalities.

As of writing the following server types are supported:

  • UDP
  • HTTP
  • Websocket (WebSocketIO)

The class architecture looks more or less like this:

  • TNJCustomServer
    • TNJUDPService
      • TNJUDPServer
    • TNJHTTPServer
    • TNJWebSocketServer
  • TNJCustomClient
    • TNJUDPService
      • TNJUDPClient
    • TNJHttpClient
    • TNJWebSocketClient

Clients and servers have traditional auxillary objects, such as request and response objects:

  • TNJServerChildClass
    • TNJCustomServerRequest
      • TNJHttpRequest
    • TNJCustomServerResponse
      • TNJHttpResponse

The following units make up the SmartNJ namespace today:

  • SmartNJ.Filesystem
  • SmartNJ.Network.Bindings
  • SmartNJ.Network
  • SmartNJ.Server
  • SmartNJ.Server.Http
  • SmartNJ.Server.Udp
  • SmartNJ.Server.WebSocket
  • SmartNJ.Streams
  • SmartNJ.System
  • SmartNJ.Udp
  • SmartNJ.WebSocket

Unified filesystem

In the previous versions of the RTL we presented several ways to store information. We had thin wrappers over cookies, local storage and filestorage. You could also use one of the 3 database engines to store data (sandboxed).

These unit still ship with the RTL, so should you want to pick one in particular that is of course not a problem. But in order to streamline and make IO as uniform as possible, I have introduced the notion of a standard “filesystem”.

The filesystem base-class can be found (as expected) in the system.filesystem.pas file. This contains the abstract class that makes up the most common filesystem operations.

NodeJS gives you a filesystem class that has direct access to the real files and folders, so be careful when you use that. As expected this filesystem implementation can be found under SmartNJ.Filesystem.pas.

And last but not least you will find a filesystem unit for visual browser applications, under SmartCL.Filesystem.pas.

The idea here is that you can write the exact same code and it will work regardless of which platform you run on.The same code you write for node.js will then save or load data in the browser as well.

This system will be expanded to include Cordova Phonegap, NodeWebkit (a hybrid post-compiler to make desktop applications) and embedded platforms.

Note: NodeJS is special in that it gives you both an asynchronous filesystem and blocking filesystem. If you want to use the blocking version, simply access it via the blocking interface (see unit file). But we strongly suggest that you avoid this since that will have a huge impact on performance. You must never use the blocking IO interface on servers, expect perhaps during startup – it will completely undermine the node.js runtime scheme and yield terrible performance.

Database support

This has been a pressing issue for quite some time, and while we are close, we still need to add more infrastructure before we can wrap the engines we want. Although classes will be available soon – being able to visually hook up data in common RAD style is what we want.

For that to happen I need the following features in the IDE:

  • Datamodules
  • Designer drop-zone for non-visual components
  • Design support for property mapping

Without these features database support under node.js will be flawed at best. But fret not, it is being worked on and we are presently looking at the following engines:

Client side we will provide the same DB API, but here it will wrap the existing engines:

  • WebSQL [deprecated by W3C]
  • SQLite [default]
  • TW3Dataset
  • IndexDB

The reason this is more complicated than it may look is first of all because IndexDB and MongoDB are object based databases, not SQL based. So the architecture really needs property mapping and the IDE to back it up. Naturally it can be done in pure code (which we have started) but the IDE with the built-in database designer is what will make this more sane to work with.

Action support

Yes, finally it is here. TCustomAction and TAction has been added to the RTL, and appropriate controls now expose an Action property that you can assign to.

Again we need the next update before this becomes more easy to work with. You need to be able to drop a TW3ActionManager on your form or datamodule, and then assign that via the property inspector to the control.

The IDE will automatically generate the “glue” code on compile for you, and it will execute as a part of your {$I ‘form: impl’} section and set things up.

Right now you have to create and assign actions manually — but at least it’s finally in place. Which is really good because that allows me to write default-actions for various tasks just like you have in Delphi, C++ builder and Lazarus.

Final words

This has been a preview, not the full load. There are many other changes to the codebase, some of the changes quite radical. Hopefully you find this interesting and will enjoy using it as much as we have enjoyed creating it.

Things that havent been covered yet are:

  • Pixi.js
  • Plumb.js
  • Stacktrace support
  • Codec framework
  • eSpeak
  • Vector graphics display control
  • Support for Scetchpad 3d models
Loading HD 3d models into your smart applications is now possible (and easy). Stills doesnt do this model justice ..

Loading HD 3d models into your smart applications is now possible (and easy). Stills doesnt do this model justice ..

And yes, I have also added support for asset management. This wont be fully visible before the IDE update, but being able to pre-load pictures, sounds and resources before your application starts is now simplicity itself. As a bonus we also added support for Require() which is more or less the de-facto JavaScript way of loading assets (and more) these days. You even get a sweet callback when its done or when something goes wrong.

Stay tuned! I’m working as fast as I can

/Jon

Smart Pascal: Cool tricks for better code

February 2, 2017 1 comment

What is the most costly operation for a HTML5 application? While there are many to pick from the most time-consuming tasks are without a doubt sizing elements and creating elements.

Both of these operations are time-consuming (in cpu terms) because they have a direct impact on the entire layout. In other words, when you create a visible element (which happens when a Smart visual control is created), the entire DOM is affected. The same naturally happens when you adjust the size of an element, because the browser will go through all it’s nodes and re-build its calculated stylesheet (which is a hidden stylesheet that the browser amalgams together).

Since this is the case it makes sense to try to avoid sizing controls as much as possible. You can’t completely avoid it of course – but the less change to width and height the better.

Use the percentages

The Smart Pascal RTL follows the traditional, Delphi and LCL esquire component model. This means that the position and width or height of a control is measured in pixels. But HTML, as you no doubt are aware of, can also work in percentages. This is a lot faster since the browser completely takes care of sizing.

A neat trick you can use is to alter the size of a child control using percentages – and this way avoid any manual calls to Resize(). It all depends on the situation of course, but if (for example) you have a child control that should always be the total width of its parent – you can in fact directly set the width to 100%.

Since the width and height properties are managed by the RTL you want to avoid altering those. They expect values to be in pixels, so changing these values can result in unexpected side-effects. But you can modify the minimum and maximum size styles without affecting the RTL.

w3_setStyle(FContent.Handle, 'min-width', '100%');
w3_setStyle(FContent.Handle, 'min-height', '100%');

The above code takes a child control (“FContent” in this example) and forces it to cover 100% of the parent’s surface. This will work as long as the actual width property does not exceed 100%. If your container is 200 pixels wide, the above code will work fine unless you manually change width to be 201 or more.

I actually use this quite often, especially when I create any form of lists, listboxes or menu systems. Normally the child items is expected to cover the whole width of the parent (a row in a grid for example), with variable height. In that case I can just adjust the height and leave the width to the browser.

Pre calculate content

Like i mentioned above the most time-consuming tasks are size-changes and creation of elements. Of the two, creation of controls is by far the most time consuming for the document object model.

While the RTL gives you controls that are more or less compatible with Delphi or LCL, they have the downside of being quite heavy. There is a lot of code involved which gives the components great depth – but that depth comes at a cost.

What the browser does really fast however, is to create elements in bulk. Just stop and think about it for a while. If creating elements is so time-consuming – then why does pages appear almost instantly? Well, if you examine the webkit HTML renderer you will discover that parsing and creating elements en-mass is highly optimized. This is sadly not the case for the createElement() function that Smart uses to create components at runtime.

What this means is that its faster to create 1000 child elements as raw HTML and inject the text into the DOM – than it is to create 1000 controls. And not just fast: extremely fast!

But then we face a problem, namely that our RTL does something very useful for us: it keeps track of handles for each element and exposes the functionality as object pascal. If we just dump in a ton of HTML then how are we going to locate, use and manipulate our child elements?

A thin wrapper

This is where I tend to use a thin wrapper. This is a class designed to introduce as little code as possible – and only expose the underlying functionality of the document object model. The DOM is actually quite rich in functions even though they can be intimidating at first.

Here is a thin wrapper I use quite a lot.

unit SmartCL.ThinWrapper;

interface

uses
  System.Types, System.Colors,
  SmartCL.System, SmartCL.Graphics, SmartCL.Components,
  SmartCL.Fonts, SmartCL.Borders;

type

  TElementArray = class external
  protected
    function GetItems(const Index : integer) : TControlHandle; external array;
  public
    property length: integer;
    property items[const Index : integer] : TControlHandle read GetItems; default;
  end;

  TWrappedElement = class
  private
    FHandle:  TControlHandle;
  protected
    function  ValueToInt(const Value: variant): integer;
    function  IntToPixels(const Value: integer): string;
    function  GetColor: TColor;
    procedure SetColor(const NewColor: TColor);
  public

    class function GetElementById(const Parent: TControlHandle;
              const ChildId: string): TControlHandle; overload;

    class function GetElementsByType(const Parent: TControlHandle;
              TagName: string;
              var Items: array of TControlHandle): boolean; overload;

  public
    property  Handle: TControlHandle read FHandle;

    property  Id: string
              read  ( FHandle.id )
              write ( Fhandle.id := Value );

    property  Title: string
              read  ( FHandle.title )
              write ( FHandle.title := Value );

    // Offset
    property  OffsetLeft: integer
              read  (ValueToInt(FHandle.offsetLeft))
              write (FHandle.offsetLeft := IntToPixels(Value));

    property  OffsetTop: integer
              read  (ValueToInt(FHandle.offsetTop))
              write (FHandle.offsetTop := IntToPixels(Value));

    property  OffsetWidth: integer
              read  (ValueToInt(FHandle.offsetWidth))
              write (FHandle.offsetWidth := IntToPixels(Value));

    property  OffsetHeight: integer
              read  (ValueToInt(FHandle.offsetHeight))
              write (FHandle.offsetHeight := IntToPixels(Value));

    // Scroll
    property  ScrollLeft: integer
              read  (ValueToInt(FHandle.scrollLeft))
              write (FHandle.scrollLeft := IntToPixels(Value));

    property  ScrollTop: integer
              read  (ValueToInt(FHandle.scrollTop))
              write (FHandle.scrollTop := IntToPixels(Value));

    property  ScrollWidth: integer
              read  (ValueToInt(FHandle.scrollWidth))
              write (FHandle.scrollWidth := IntToPixels(Value));

    property  ScrollHeight: integer
              read  (ValueToInt(FHandle.scrollHeight))
              write (FHandle.scrollHeight := IntToPixels(Value));

    // Client
    property  clientLeft: integer
              read  (ValueToInt(FHandle.clientLeft))
              write (FHandle.clientLeft := IntToPixels(Value));

    property  clientTop: integer
              read  (ValueToInt(FHandle.clientTop))
              write (FHandle.clientTop := IntToPixels(Value));

    property  clientWidth: integer
              read  (ValueToInt(FHandle.clientWidth))
              write (FHandle.clientWidth := IntToPixels(Value));

    property  clientHeight: integer
              read  (ValueToInt(FHandle.clientHeight))
              write (FHandle.clientHeight := IntToPixels(Value));

    // Node
    property  NodeName: string read (FHandle.nodeName);
    property  NodeType: string read (FHandle.nodeType);
    property  NodeValue: variant
              read  (FHandle.nodeValue)
              write (FHandle.nodeValue := Value);

    property  Children: TElementArray read ( TElementArray(FHandle.children) );

    property  InnerHTML: string
              read  ( Handle.innerHTML )
              write ( Handle.innerHTML := Value);

    property  InnerText: string
              read  ( Handle.innerText )
              write ( Handle.innerText := Value);

    property  Color: TColor read GetColor write SetColor;

    function  Wrap(const Handle: TControlHandle): TWrappedElement;

    function  GetElementById(const Id: string): TControlHandle; overload;

    function  GetElementsByType(const TagName: string;
              var Items: array of TControlHandle): boolean; overload;

    procedure Click;

    constructor Create(TagId: string); overload; virtual;
    constructor Create(Parent: TControlHandle; TagId: string); overload; virtual;
    constructor Create(TagHandle: TControlHandle); overload; virtual;
  end;

implementation

//#############################################################################
// TWrappedElement
//#############################################################################

constructor TWrappedElement.Create(TagId: String);
begin
  inherited Create;
  FHandle := BrowserAPI.Body.GetChildById(TagId);
end;

constructor TWrappedElement.Create(TagHandle: TControlHandle);
begin
  inherited Create;
  FHandle := TagHandle;
end;

constructor TWrappedElement.Create(Parent: TControlHandle; TagId: string);
begin
  inherited Create;
  FHandle := Parent.GetChildById(TagId);
end;

procedure TWrappedElement.Click;
begin
  FHandle.click();
end;

function TWrappedElement.Wrap(const Handle: TControlHandle): TWrappedElement;
begin
  result := TWrappedElement.Create( Handle );
end;

function TWrappedElement.IntToPixels(const Value: integer): string;
begin
  result := Value.ToString() + 'px';
end;

function TWrappedElement.ValueToInt(const Value: variant): integer;
begin
  asm
    if (@Value)
    {
      if (typeof(@Value) === "number") {
        @result = @Value
      } else {
        if (typeof(@Value) === "string")
        {
          @Value = parseInt(@Value);
          if (!isNaN(@Value))
            @result = @Value;
        }
      }
    } else {
      @result = 0;
    }
  end;
end;

function TWrappedElement.GetColor: TColor;
begin
  if (FHandle) then
  result := StrToColor( w3_getStyleAsStr(FHandle, 'backgroundColor') ) else
  result := clNone;
end;

procedure TWrappedElement.SetColor(const NewColor: TColor);
begin
  if (FHandle) then
  begin
    if NewColor <> clNone then
    FHandle.style.backgroundColor := ColorToWebStr(NewColor) else
    FHandle.style.backgroundColor := 'transparent';
  end;
end;

class function TWrappedElement.GetElementsByType(const Parent: TControlHandle;
         TagName: string;
         var Items: array of TControlHandle): boolean;
begin
  if (parent) then
  begin
    asm
      @items = (@Parent).getElementsByTagName(@TagName);
    end;
  end else
  Items.Clear();
  result := Items.Count > 0;
end;

function TWrappedElement.GetElementsByType(const TagName: string;
         var Items: array of TControlHandle): boolean;
begin
  if (FHandle) then
  begin
    asm
      @items = (@FHandle).getElementsByTagName(@TagName);
    end;
  end else
  Items.Clear();
  result := Items.Count > 0;
end;

class function TWrappedElement.GetElementById(const Parent: TControlHandle;
         const ChildId: string): TControlHandle;
begin
  if (Parent) then
  begin
    result := Parent.getElementById(ChildId);
    if not (result) then
      result := Parent.getElementById( ChildId.ToLower().Trim() );
  end else
  result := unassigned;
end;

function TWrappedElement.GetElementById(const Id: string): TControlHandle;
begin
  if Id.Length > 0 then
  begin
    result := FHandle.getElementById(Id);
    if not (result) then
      result := FHandle.getElementById( Id.ToLower().Trim() );
  end else
  result := unassigned;
end;

end.

If you are pondering how on earth is that going to help, here is how it works.

All Smart controls are simply JavaScript code designed to manage a real, underlying HTML element. The default element type TW3CustomControl creates is DIV. Controls like TW3TextBox overrides the function that creates this element and replace that with an input element instead. And other controls do the same.

So just because something is not visible to a fully blown TW3CustomControl, doesn’t mean it’s not there. And by using the wrapper we can easily hook rouge or non classified html elements without creating them.

Let’s for example say you inject a bit of HTML into a panel, like this:

procedure TForm1.MakeHTML;
var
  x: integer;
  LHtml: string;
begin
  // make X number of div items
  for x:=1 to 100 do
  begin
    LHtml += Format('
<div id="obj%d">Item #%d</div>
', [x,x]);
  end;

  // inject into our panel
  W3Panel1.InnerHTML := LHtml;
end;

To access one of these DIV elements (which we now have 100 of), we can use our thin wrapper to make it programatically easier:

procedure TForm1.MakeHTML;
var
  x: integer;
  LHtml: string;
  LItem: TWrappedElement;
begin
  // make X number of div items
  for x:=1 to 100 do
  begin
    LHtml += Format('
<div id="obj%d">Item #%d</div>
', [x,x]);
  end;

  // inject into our panel
  W3Panel1.InnerHTML := LHtml;

  // Create wrapper for item #12
  // We pass the handle to the parent (which is the form)
  // and the name. The class will find the element
  LItem := TWrappedElement.Create(W3Panel1.Handle, 'obj12');
end;

Voila! LItem can now be used just like any other Smart control. But remember that this is a thin wrapper, meaning that you are actually digging into the document object model directly.

I should mention a few words about the browser’s calculated stylesheet. One of the things you are going to notice is that width and height is not always going to be correct. This is not due to our code, but because the browser always regards your values as proposals, not absolutes.

So even if you set a control to say, 400px in width – depending on the situation the browser may find it more suitable to set 389px instead. And if you make the mistake of reading it back from the stylesheet – it will report 400px. But this is because the stylesheet is regarded as a proposal.

What you need to do is to write to the stylesheet, but read from the calculated styles. This is why the Smart RTL calls the w3_getStyleAsInt() function when reading these values.

Just so you know in case you think the wrapper is reporting wrong values. It’s not. The browser just works differently because of CSS. Cascading means that styles will merge together, so a button can have 50 gradients assigned to it – and they will all be amalgamated into one and represented in the calculated stylesheet as a single whole.

Other tricks

I could go on for days but I think these two will be handy for now. I would urge you to examine and learn how to use the GetElementByType() and GetElementById() so you get familiar with navigating the DOM like that. GetElementByType() is really cool – it allows you to extract a list of items based on type.

So to get all the DIV elements you can simply do:

var LDivs := LItem.GetElementByType(W3Panel1.Handle, 'div');

I should also mention that a thin-wrapper is only as good as you make it. The code above was never made to do the same as TW3CustomControl can. There is no cosy class wrapping of fonts, constraints, borders or reading of style properties. To enjoy these high-level RTL functions you will have to stick to the RTL.

But, for cases where you want to pre-generate relatively simple elements, like rows in a listbox or some form of menu items – a thin wrapper can mean the difference between useless and brilliant.

Oh, you might be interested in a “Styles” wrapper. I have only fleshed out a handfull of properties, but it does make low-level access a lot easier. If you finish it PM me. The documentation can be found here: http://www.w3schools.com/jsref/dom_obj_style.asp

  TWrappedStyle = class external
  public
    // stretch|center|flex-start|flex-end|space-between|space-around|initial|inherit
    property  alignContent: string;

    // stretch|center|flex-start|flex-end|baseline|initial|inherit
    property  alignItems: string;

    // auto|stretch|center|flex-start|flex-end|baseline|initial|inherit
    property  alignSelf: string;

    // http://www.w3schools.com/jsref/prop_style_animation.asp
    property  animation: string;

    // time|initial|inherit
    property  animationDelay: string;

    // normal|reverse|alternate|alternate-reverse|initial|inherit
    property  animationDirection: string;

    // time|initial|inherit
    property  animationDuration: string;

    // none|forwards|backwards|both|initial|inherit
    property  animationFillMode: string;

    // number|infinite|initial|inherit
    property  animationIterationCount: string;

    // none|keyframename|initial|inherit
    property  animationName: string;

    // linear|ease|ease-in|ease-out|cubic-bezier(n, n, n, n)|initial|inherit
    property  animationTimingFunction: string;

    // running|paused|initial|inherit
    property  animationPlayState: string;

    // color image repeat attachment position size origin clip|initial|inherit
    property background: string;

    // scroll|fixed|local|initial|inherit
    property backgroundAttachment: string;

    // color|transparent|initial|inherit
    property backgroundColor: string;

    // url('URL')|none|initial|inherit
    property backgroundImage: string;

    // http://www.w3schools.com/jsref/prop_style_backgroundposition.asp
    property backgroundPosition: string;

    // repeat|repeat-x|repeat-y|no-repeat|initial|inherit
    property backgroundRepeat: string;

    // border-box|padding-box|content-box|initial|inherit
    property backgroundClip: string;

    // padding-box|border-box|content-box|initial|inherit
    property backgroundOrigin: string;
  end;

Well, more and more tricks will surface, so stay tuned guys!

Smart Pascal: BTree storage anyone?

February 1, 2017 Leave a comment

btreeDictionaries are cool but they are only as good as the mechanisms supporting them. So I figured I could see if we could get more bang for the buck with a dedicated BTree class for Smart Pascal.

If you are wondering what on earth a BTree routine is, head over to Wikipedia and gander up on the technical side here: https://en.wikipedia.org/wiki/B-tree. In short it allows you to store some data connected to a string. Actually, its connected to a value identifier – but we can do a checksum of a string and use that as the key. So dictionary. Sort of.

What is special about the Smart Pascal version? Well, for one it doesn’t use pointers. And secondly you can save it to a stream. Typically classes like this don’t ship with a SaveToStream() method because it’s mostly intended to be used at runtime. But JavaScript have a few perks that force us to think differently.

Hope you enjoy it!


unit BTree;

interface

uses
  System.Types,
  System.Types.Convert,
  System.JSON,
  System.NameValuePairs,
  System.Streams,
  System.Stream.Writer,
  System.Stream.Reader;

type

  TBTreeNode = class(JObject)
  public
    Ident: integer;
    Data: variant;
    Left: TBTreeNode;
    Right: TBTreeNode;
  end;

  TBTreeFileItem = record
    fiIdent:  integer;
    fiData: variant;
  end;

  TBTreeProcess = procedure (const Node: TBTreeNode; var Cancel: boolean);

  TBTree = class(TObject)
  private
    FRoot:  TBTreeNode;
    FCurrent: TBTreeNode;
  public
    property  Root: TBTreeNode read FRoot;
    property  Empty: boolean read ( FRoot = nil );

    function  Add(const Ident: integer; const Data: variant): TBTreeNode;overload;virtual;
    function  Add(const Ident: string; const Data: variant): TBTreeNode;overload;virtual;

    function  &Contains(const Ident: integer): boolean;overload;virtual;
    function  &Contains(const Ident: string): boolean;overload;virtual;

    function  Remove(const Ident: integer): boolean;overload;virtual;
    function  Remove(const Ident: string): boolean;overload;virtual;

    function  Read(const Ident: integer): variant;overload;virtual;
    function  Read(const Ident: string): variant;overload;virtual;

    procedure Write(const Ident: string; const NewData: variant);overload;virtual;
    procedure Write(const Ident: integer; const NewData: variant);overload;virtual;

    procedure Clear;overload;virtual;
    procedure Clear(const Process: TBTreeProcess);overload;virtual;

    function  ToArray: TVarArray;
    function  ToString: string;
    function  Size: integer;

    function  ToJSON: string;
    procedure FromJSON(const Data: string);

    procedure SaveToStream(const Stream: TStream);
    procedure LoadFromStream(const Stream: TStream);

    procedure ForEach(const Process: TBTreeProcess);

    constructor Create;
  end;


implementation


(* These are the IO signatures used for storage *)
const
CNT_BTREE_STREAM_HEADER = $BABE0001;
CNT_BTREE_ITEM_HEADER = $0001BABE;

//#############################################################################
// TBTree
//#############################################################################

constructor TBTree.Create;
begin
  inherited Create;
  FRoot := nil;
  FCurrent := nil;
end;

procedure TBTree.Clear;
begin
  FCurrent := nil;
  FRoot := nil;
end;

procedure TBTree.Clear(const Process: TBTreeProcess);
begin
  ForEach(Process);
  Clear;
end;

function TBTree.ToJSON: string;
begin
  if FRoot<>nil then
    result := Json.stringify(FRoot);
end;

procedure TBTree.FromJSON(const Data: string);
begin
  if not empty then
    Clear;
  FRoot := TBTreeNode( JSON.Parse(data) );
end;

procedure TBTree.SaveToStream(const Stream: TStream);
var
  LWriter: TStreamWriter;
  LData: Array of TBTreeFileItem;
  LRaw: TByteArray;
  LItem: TBTreeFileItem;
begin

  (* First, cache up all the data in an array. We need to do this
     in order to write the node-count on top of the file *)
  ForEach( procedure (const Node: TBTreeNode; var Cancel: boolean)
  begin
    LItem.fiIdent := Node.Ident;
    LItem.fiData := Node.Data;
    LData.add(LItem);
  end);

  LWriter := TStreamWriter.Create(Stream);
  try
    (* Write the magic identifier for the file *)
    LWriter.WriteInteger(CNT_BTREE_STREAM_HEADER);

    (* Write the number of items in the file *)
    LWriter.writeinteger(LData.Count);

    (* Now write each item *)
    for LItem in LData do
    begin
      (* Convert variant to byte-array *)
      LRaw := TDataType.VariantToBytes(LItem.fiData);

      (* Write the identifier for the record *)
      LWriter.WriteInteger(CNT_BTREE_ITEM_HEADER);

      (* Write the item-id *)
      LWriter.WriteInteger(LItem.fiIdent);

      (* Write the # of bytes in the data section *)
      LWriter.WriteInteger(LRaw.Count);

      (* Write the data section *)
      LWriter.Write(LRaw);
    end;
  finally
    LWriter.free;
  end;
end;

procedure TBTree.LoadFromStream(const Stream: TStream);
var
  LReader: TStreamReader;
  LHead: integer;
  LId: integer;
  LBytes: integer;
  LRaw: TByteArray;
  LValue: variant;
  LCount: integer;
begin
  (* Flush content if not empty *)
  if not Empty then
    Clear;

  (* Setup the reader *)
  LReader := TStreamReader.Create(Stream);
  try

    (* Validate the header *)
    LHead := LReader.ReadInteger;
    if LHead = CNT_BTREE_STREAM_HEADER then
    begin
      (* Get the count *)
      LCount := LReader.ReadInteger;

      while LCount>0 do
      begin
        LHead := LReader.ReadInteger;
        if LHead = CNT_BTREE_ITEM_HEADER then
        begin
          (* Read the identifier *)
          LId := LReader.ReadInteger;

          (* read the # of bytes for the variant *)
          LBytes := LReader.ReadInteger;

          (* Flush any lingering data *)
          LRaw.Clear;

          (* Read the raw data that makes up the variant *)
          if LBytes>0 then
          begin
            LRaw := LReader.read(LBytes);

            (* Convert from bytes to intrinsic *)
            LValue := TDatatype.BytesToVariant(LRaw);

            (* Add to tree *)
            self.Add(Lid, LValue);
          end;

        end else
        raise EW3Exception.CreateFmt('Invalid item header, expected %d not %d',
        [CNT_BTREE_ITEM_HEADER,LHead]);

        dec(LCount);
      end;

    end else
    raise EW3Exception.CreateFmt('Invalid stream header, expected %d not %d',
    [CNT_BTREE_STREAM_HEADER,LHead]);

  finally
    LReader.free;
  end;
end;

function TBTree.Size: integer;
var
  LCount: integer;
begin
  ForEach( procedure (const Node: TBTreeNode; var Cancel: boolean)
    begin
      inc(LCount);
    end);
  result := LCount;
end;

function TBTree.ToArray: TVarArray;
var
  Data: TVarArray;
begin
  ForEach( procedure (const Node: TBTreeNode; var Cancel: boolean)
    begin
      Data.add(Node.Ident);
    end);
  result := data;
end;

function TBTree.ToString: string;
begin
  for var x in ToArray do
  begin
    result += TVariant.AsString(x) + #13;
  end;
end;

function TBTree.Add(const Ident: string; const Data: variant): TBTreeNode;
begin
  result := Add( TString.CalcCRC(Ident), Data);
end;

function TBTree.Add(const Ident: integer; const Data: variant): TBTreeNode;
var
  LNode:  TBTreeNode;
begin
  LNode := new TBTreeNode;
  LNode.Ident := Ident;
  LNode.Data := data;

  if (FRoot = nil) then
  FRoot := LNode;

  FCurrent := FRoot;

  while (true) do
  begin
    if (Ident < FCurrent.Ident) then
    begin
      if (FCurrent.left = nil) then
      begin
        FCurrent.left := LNode;
        break;
      end else
      FCurrent := FCurrent.left;
    end else
    if (Ident > FCurrent.Ident) then
    begin
      if (FCurrent.right = nil) then
      begin
        FCurrent.right := LNode;
        break;
      end else
      FCurrent := FCurrent.right;
    end else
    break;
  end;
  result := LNode;
end;

function TBTree.Read(const Ident: string): variant;
begin
  result := Read( TString.CalcCRC(Ident) );
end;

function TBTree.Read(const Ident: integer): variant;
begin
  Result := unassigned;
  FCurrent := FRoot;
  while (FCurrent <> nil) do
  begin
    if (Ident < FCurrent.Ident) then
    FCurrent := Fcurrent.left else
    if (Ident > Fcurrent.Ident) then
    FCurrent := FCurrent.Right else
    begin
      result := FCUrrent.Data;
      break;
    end
  end;
end;

procedure TBTree.Write(const Ident: string; const NewData: variant);
begin
  Write(TString.CalcCRC(Ident), NewData);
end;

procedure TBTree.Write(const Ident: integer; const NewData: variant);
begin
  FCurrent := FRoot;
  while (FCurrent <> nil) do
  begin
    if (Ident < FCurrent.Ident) then
    FCurrent := Fcurrent.left else
    if (Ident > Fcurrent.Ident) then
    FCurrent := FCurrent.Right else
    begin
      FCurrent.Data := NewData;
      break;
    end
  end;
end;

function  TBTree.&Contains(const Ident: string): boolean;
begin
  result := &Contains(TString.CalcCRC(Ident));
end;

function TBTree.&Contains(const Ident: integer): boolean;
begin
  Result := false;
  if FRoot <> nil then
  begin
    FCurrent := FRoot;

    while ( (not Result) and (FCurrent <> nil) ) do
    begin
      if (Ident < FCurrent.Ident) then
      FCurrent := Fcurrent.left else

      if (Ident > Fcurrent.Ident) then
        FCurrent := FCurrent.Right else
      begin
        Result := true;
      end
    end;
  end;
end;

function TBTree.Remove(const Ident: string): boolean;
begin
  result := Remove(TString.CalcCRC(Ident));
end;

function TBTree.Remove(const Ident: integer): boolean;
var
  LFound: boolean;
  LParent: TBTreeNode;
  LChildCount: integer;
  LReplacement,
  LReplacementParent: TBTreeNode;
begin
  LFound := false;
  LParent := nil;
  FCurrent := FRoot;

  while (not LFound) and (FCurrent<>nil) do
  begin
    if (Ident < FCurrent.Ident) then
    begin
      LParent := FCurrent;
      FCurrent:= FCurrent.left;
    end else
    if (Ident > FCurrent.Ident) then
    begin
      LParent := FCurrent;
      FCurrent := FCurrent.right;
    end else
    begin
      LFound := true;
    end;

    if (LFound) then
    begin
      LChildCount:=0;
      if (FCurrent.left<>nil) then inc(LChildCount);
      if (FCurrent.right<>nil) then inc(LChildCount);
      //LChildCount := (if FCurrent.left <> nil then 1 else 0) + (if FCurrent.right <> nil then 1 else 0);
      if (FCurrent = FRoot) then
      begin
        case (LChildCOunt) of
        0: FRoot := nil;
        1: FRoot := if FCurrent.right = nil then FCurrent.left else FCurrent.Right;
        2: begin

            LReplacement := FRoot.left;
            while (LReplacement.right <> nil) do
            begin
              LReplacementParent := LReplacement;
              LReplacement := LReplacement.right;
            end;

            if (LReplacementParent <> nil) then
            begin
              LReplacementParent.right := LReplacement.Left;
              LReplacement.right := FRoot.Right;
              LReplacement.left := FRoot.left;
            end else
            LReplacement.right := FRoot.right;
          end;
        end;

        FRoot := LReplacement;
      end else
      begin
        case LChildCount of
        0:  if (FCurrent.Ident < LParent.Ident) then
            Lparent.left  := nil else
            LParent.right := nil;
        1:  if (FCurrent.Ident < LParent.Ident) then
            begin
              if (FCurrent.Left = NIL) then
              LParent.left := FCurrent.Right else
              LParent.Left := FCurrent.Left;
            end else
            begin
              if (FCurrent.Left = NIL) then
              LParent.right := FCurrent.Right else
              LParent.right := FCurrent.Left;
            end;
        2:  begin
              LReplacement := FCurrent.left;
              LReplacementParent := FCurrent;

              while (LReplacement.right <> nil) do
              begin
                LReplacementParent := LReplacement;
                LReplacement := LReplacement.right;
              end;
              LReplacementParent.right := LReplacement.left;

              LReplacement.right := FCurrent.right;
              LReplacement.left := FCurrent.left;

              if (FCurrent.Ident < LParent.Ident) then
              LParent.left := LReplacement else
              LParent.right := LReplacement;
            end;
          end;
        end;
      end;
  end;

  result := LFound;
end;

procedure TBTree.ForEach(const Process: TBTreeProcess);

  function ProcessNode(const Node: TBTreeNode): boolean;
  begin
    (* Default to false. If true is defined here, the operation
       has been canceled by the user *)
    result := false;

    if (Node <> nil) then
    begin

      (* Process left path first *)
      if (node.left <> nil) then
      begin
        result := ProcessNode(Node.left);
        if result then
        exit;
      end;

      (* process midpoint *)
      Process(Node, result);
      if result then
      exit;

      (* current right path *)
      if (Node.right <> nil) then
      begin
        result:=ProcessNode(Node.right);
        if result then
        exit;
      end;
    end;
  end;

begin
  ProcessNode(FRoot);
end;

end.

Smart Pascal: Update specifics

January 20, 2017 Leave a comment

With what is probably the biggest update Smart Mobile Studio has seen to date only a couple of weeks away, what should the Smart Pascal programmer know about the new RTL?

I have written quite a few articles on the changes, so you may want to backtrack and look at them first – but in this post I will try to summerize two of the most important changes.

Unit files, a background

The first version of Smart Mobile Studio (sms) was designed purely for webkit based, HTML5 applications. As the name implies the initial release was designed purely for mobile application development; with the initial target firmly fixed on IPhone and IPad. As a result, getting rid of webkit exclusive code has been very time-consuming to say the least.

At the same time we added support for embedded devices, like the Espruino micro-controller, and in leu of that process -it became clear that the number of devices we can support is huge. As of writing we have code for more than 40 embedded devices lined up, ready to be added to the RTL (after the new RTL is in circulation).

Add to this the fact that node.js is now the most powerful cloud technology in the world, the old “DOM only”, webkit oriented codebase was hopelessly narrow minded and incapable of doing what we wanted. Something had to give, otherwise this would be impossible to maintain – not to mention document in any reasonable logical form.

A better unit system

I had to come up with a system that preserved universal code – yet exposed special features depending on the target you compile for.

So when you compile for HTML5 you get all the cool stuff the document object model (DOM) has to offer, but also full access to the universal units. Same goes for node.js, its has all the standard stuff that is universal – but expose a ton of JS objects that exist nowhere else.

What I ended up with was that each namespace should have the same units, but adapted for the platform it represents. So you will have units like System.Time.pas, which represents universal functionality regarding timing and synchronization; But you are supposed to use the one prefixed by your target namespace. For instance, SmartCL.Time or SmartNJ.Time. In cases where the platform doesnt expose anything beyond the universal code, you simply fall back on the System.*.pas unit.

namespace_scheme

It may look complex, but it’s super easy once you get it

The rules are super simple:

  • Coding for the browser? Use the units prefixed with SmartCL.*
  • Coding for node.js? Use the units prefixed with SmartNJ.*
  • Coding for embedded? Use the units prefixed with SmartIOT.*

Remember that you should always include the system version of the unit as well. So if you include SmartCL.Time.pas, add System.Time.pas to the uses list as well. It makes it easier to navigate should you want to CTRL + Click on a symbol.

Better synchronization engine

JavaScript is async by nature. Getting JavaScript to behave like traditional object pascal is not only hard, it is futile. Besides, you dont want to be treated like a child and shielded from the reality of JavaScript – because that’s where all the new cool stuff is at.

In Delphi and Lazarus we are used to our code being blocking. So if you create child objects in your constructur, these objects are available there and then. Well, thats not how JavaScript works. You can create an object, the code continues, but in reality the object is not finished yet. It may in fact be assembling in the background (!)

To make writing custom-controls sane and human, we had to change a few things. So in your constructor (which is InitializeObject in our RTL, not the actual class constructor) you are expected to create child object – but you should never use them.

Instead there is a method called ObjectReady() which naturally fires when all your child elements have been constructed properly and the control has been safely injected into the document object model. This is where you set properties and start to manipulate your child objects.

synchro

So finally the setup process is stable, fast and working as you expect. As long as you follow this simple rule, your controls will appear and function exactly like you expect them to.

  • Never override Create and Destroy, use InitializeObject and FinalizeObject
  • Override ObjectReady() to set initial values

Creation flags

We have also added creation flags to visual controls. This is a notion familiar to both Delphi’s VCl and Lazarus LCL component structures. It allows you to define a few flags that determines behavior.

For instance, if your control is sized manually and will never really need to adapt – then it’s a waste of time to have Resize() fire when the size is altered. Your code will run much faster if the RTL can just ignore these changes. The flags you can set are:

  TW3CreationFlags = set of
    (
    cfIgnoreReadyState,     // Ignore waiting for readystate
    cfSupportAdjustment,    // Controls requires boxing adjustment (default!)
    cfReportChildAddition,  // Dont call ChildAdded() on element insertion
    cfReportChildRemoval,   // Dont call ChildRemoved() on element removal
    cfReportMovement,       // Report movements? Managed call to Moved()
    cfReportResize,         // Report resize? Manages call to Resize()
    cfAllowSelection,       // Allow for text selection
    cfKeyCapture            // assign tabindex, causing the control to issue key and focus events
    );

To alter the default behavior you override the method CreationFlags(), which is a class function:

class function TW3TagObj.CreationFlags: TW3CreationFlags;
begin
  result := [
    // cfAllowSelection,
    cfSupportAdjustment,
    cfReportChildAddition,
    cfReportChildRemoval,
    cfReportMovement,
    cfReportReSize
    ];
end;

Notice the AlloSelection flag? This determines if mouse selection of content is allowed. If you are making a text display, or editor, or anything that the user should be allowed to select – then you want this flag set.

The cfKeyCapture is also a flag many people have asked about. It determines if keyboard input should be allowed for the element. By default this is set to OFF. But now you dont have to manually mess around with element attributes. Just add that flag and your control will fire the OnKeyPress event whenever it has focus and the user press a key.

Tweening and fallback effects

In the last update a bug managed to sneak into our SmartCL.effect.pas library, which means a lot of people havent been able to fully use all the effects. This bug has now been fixed, but we have also added a highly efficient tweening library. The concept is that effects will be implemented both as hardware acellerated, GPU powered effects – but also with CPU based tweening alternatives.

For mobile devices it is highly adviced to use the GPU versions (the normal SmartCl.Effects.pas versions).

When you add the unit SmartCL.Effects.pas to your forms, what happens is that ALL TW3MovableObject based controls are suddenly extended with around 20 fx prefixed methods. This is because we use partial classes (which means classes can be extended).

This is super powerful, and with a single line of code you can make elements fly over the screen, shrink, zoom out or scale in size. You even get a callback when the effect has finished – and you can daisy-chain calls to execute a whole list of effects in sequence.

FMyButton.fxSizeTo(10, 10, 200, 200, 0.8, procedure ()
  begin
   writeln("Effect is done!");
  end);

And execute in sequence. The effects will wait their turn and execute one by one.

FMyButton.fxMoveTo(10,10, 0.8)
  .fxSizeTo(100,100, 0.8)
  .fxFadeTo(0.3, 0.8);

Stay tuned for more info!

Smart Pascal: Node.js by example

January 16, 2017 Leave a comment

nodeThe update for the Smart Mobile Studio RTL is nearing completion. We still have a little to go, but all in all we have cleaned up the VJL considerably and made a much better organization. We now have 3 namespaces (read: groups of units), namely: System, SmartCL and SmartNJ. The latter representing the Node.js aspect of our codebase. We also have a fourth namespace in the making, namely Embedded. Currently our embedded support is limited to Espruino – but I have begun working on the Arduino and Arduino mega codebase. When these are complete they will all be in the Embedded namespace.

Ok, node.js so where do I begin. Perhaps its best to explain a few fundamental keys to how the new RTL is organized so you understand just much Smart gives you over vanilla JavaScript.

I mentioned the system namespace above, but what exactly does that mean? The system namespace contains universal code. Code that runs on all JavaScript platforms regardless of a DOM being present or not. So you get to enjoy stuff like:

  • Traditional TStream, TMemoryStream and TFileStream
  • Codec classes
  • Memory allocation and blue pointers
  • Encryption classes
  • Delayed execution (TW3Dispatch, system.time.pas)
  • Event objects
  • Filesystem
  • Datatype conversion (system.type.convert)
  • .. and the whole system namespace

Notice the TFileStream there? You will find that the system namspace contains an abstract implementation of a class called TFileSystem. And this is where things get funky. When you create a SmartCL application (that is, a visual HTML5 application), the implemented filesystem is in SmartCL.Filesystem. When you create a node.js application, the implemented filesystem can be found in SmartNJ.filesystem. See what I’m getting at here? The idea is that regardless of the runtime environment, you will always be able to use the same code no matter what engine you run on.

Ok, hopefully that gave you some context to work with, now let’s move on and create a fancy server for node.js !

Sexy server time

We are going to create a simple yet powerful UDP client / server application. I picked this because it’s sort of the odd-ball in our collection of server objects. UDP is connection-less and is ridiculously easy to use. At the same time there is no verification and assurance of delivery (that’s the flipside). But UDP is excellent for “live” logging, or for inter-service communication between micro-services on the same network or router.

Right, let’s just jump straight in:

var Server: TNJUDPServer;
Server := TNJUDPServer.Create;

Server.OnMessage := procedure (Sender: TObject; Data: variant;
  RequestInfo: TNJUDPRequestInfo)
begin
  writeln("Recieved data:" + string(data));
end;

Server.OnError := procedure (Sender: TObject;
  ErrorObj: TJsErrorObject)
begin
  writelnF("Error[%s]: %s", [variant(ErrorObj.stack).toString(), ErrorObj.message]);
end;

Server.OnClose := procedure (Sender: TObject; ErrorObj: TJsErrorObject)
begin
  writeln("Server closed");
end;

Server.OnAfterServerStarted := procedure (sender: TObject)
begin
  server.Send("Your first Smart server is now running");
end;

Server.Port := 1881;
Server.Address := '127.0.0.1';
Server.MulticastLoopback := true;
Server.Broadcast := true;
server.Exclusive:= false;
Server.Start();

Congratulations! You have just created your first node.js powered UDP server! It doesnt get much easier than this does it? Try to guess how much JS code you would have to write to get all the perks of the system namespace + the node.js namespace.

Some guy commented on an article I wrote the other day, asking me to give him a single reason why Smart Mobile Studio is better than just writing JavaScript. The answere is simple: producivity. Some of the code you get in a simple class in Smart represents hundreds or thousands of lines of JavaScript code.

What do you think is more productive? Maintaining 2 megabyte of raw JavaScript code; a binary kebab of spaghetti code that can go belly up if you missplace as much as a comma. Or the 2k of clean, easy to read, easy to understand object oriented pascal code?

I rest my case.

Right, with the server done already, let’s have a peek at the client:

LClient := TNJUDPClient.Create;
LClient.Port := 1881;
LClient.Address := "localhost";
LClient.Bindings.Add('127.0.0.1',1881);
LClient.OnAfterStarted := procedure (sender: TObject)
begin
  LClient.Send("Client is now active", 1881, 'localhost',
  procedure (ErrorObj:   TJsErrorObject)
  begin
    writeln("An error occured:" + ErrorObj.Stack.ToString());
  end);
LClient.Active := true;

And that my friend is pretty much it!

Node.js really is awesome once you get to approach it on our terms. When you come to the JavaScript virtual machine through the OOP layer that Smart Pascal gives you, it’s a whole different reality.

Write a cluster ready, platform independent client/server application in less time than it takes to boot up Visual studio

Write a cluster ready, platform independent client/server application in less time than it takes to boot up Visual studio

What is the benefit here you might ask? All of this can be done in Delphi and Lazarus without much problem. The benefit is this: the generated code is platform independent, it will run on any platform as long as node.js is installed. It will behave identical regardless of operating-system. And you can cluster, clone and replicate instances to your heart’s desire.

Node.js is also easy to host, cheap and accessible. Native hosting is expensive and requires much more work. When something goes wrong inside a 200 megabyte service hosted on Amazon.. I have been there. Having to track down the bugs, re-build the executable while your boss is going mental – spend 10 minutes just uploading the damn thing, then you have to uninstall the service, reboot the whole instance, re-install the new .exe file and hope that you did manage to catch that bug.

Compare that to fixing the bug, uploading the new JS file and then inform PM2 that you need to hot-swap the service code. It’s a whole new paradigm.

Websocket server

With UDP behind us, let’s look at something a lot more complex. And when I write complex, I dont mean for you. Websocket is the fastest growing protocol standard these days. It’s basically an extension of the ordinary HTTP protocol. It is designed for long-term connections (read: not stateless, single shot operations like http) and is full duplex and async. So the server can talk to the client and visa versa without having to wait it’s turn. Just fire off as many messages as you like, whenever you like – and it will arive in the same order on the other side.

var
LServer: TNJWebSocketServer;

LServer := TNJWebSocketServer.Create;
LServer.ClientTracking := true;
LServer.Port := 1881;

// Setup our own protocol commands
LServer.On("command1", procedure (const Socket: JWsSocket; const Data: variant)
  begin
    writeln("Command #1 executed, recieved data:");
    writeln(data);
  end);

LServer.On("command2", procedure (const Socket: JWsSocket; const Data: variant)
  begin
    writeln("Command #2 executed");
    // broadcast a "hello" response with a byte array to *ALL*
    // connected clients. Making chat applications is very easy here
    socket.Emit("Hello", [12,13,14,14]);
  end);

LServer.Active := true;

If you look closely you will notice the use of the On keyword. This is what we use to define our server-side protocol. It’s essentially trigger words that you associate with a piece of code. When the server recieves a message it will read the message-name, look up a associated code and execute it. But let’s look at the background for this so you dont end up downloading the wrong package.

Normally you install the node.js websocket package first, and then you install something called socket.io on top of that. The On keyword we used above is actually not a part of the default websocket standard (which is more low level). This functionality is provided by a separate package called socket.io.

But, since most people install and use websocket just to use socket.io, it makes sense to merge these two packages into a single, stand-alone distro that gives you everything in one go. And this package is cleverly named Websocket-IO (it’s just knockout names isnt it).

I usually install node packages in my project output folder

I usually install node packages in my project output folder

If you want to read more about websocket-io, head over to the npm website and have a peek at: https://www.npmjs.com/package/nodejs-websocket

Client: Needless to say, the client class is just as simple so I dont really see a point in repeating that. In fact, all servers and clients inherit from the same base-class.

What should be stressed is that websocket is easy to use from Delphi as well. So if you have a native client and want to talk with your node.js server, I strongly suggest you stay clear of REST (which is one of the most wasteful protocols ever invented) and instead use websocket.

Note: since websocket is an extension to http, it is perfectly safe to use in a commercial environment. REST requires a lot more CPU and generates much more data on the network than websocket. The most time consuming and intensive aspects of a http call is during connection, and websocket is based on async and full duplex communication, with message caching and more handled automatically – so there is no reason why you should chose REST over websocket.

Websocket allows you to broadcast messages, or to filter messages based on criteria. Messaging and data-flow protocols are very easy to implement

Websocket allows you to broadcast messages, or to filter messages based on criteria. Messaging and data-flow protocols are very easy to implement

When you add the fact that websocket is also allowed from the browser, you can offer your services to the Javascript community without any extra development. So you get to cater for both native and JS clients. Being able to connect and interact with your custom servers directly from the browser opens up new possebilities. Want to display live information? With a full duplex connection the server can now inform you whenever something has changed instead of your clients polling on interval.

A Delphi extension to websocket (for Indy) was implemented and released by Andre Mussche a while back. The code is super easy to work with and bridges the world of Smart Pascal and Delphi quite nicely:
https://github.com/andremussche/DelphiWebsockets

If you need information about how to use websocket regardless of language, just google the topic. There are thousands of resources out there.

Last but not least

I hope you have found this super simple introduction to node.js server coding inspirational. Node.js is a huge topic and there are roughly 350.000 different packages available online for it. Writing wrapper classes for node packages is not hard either. You do have to know your way around object pascal and Javascript, but if you look at my units and how we have solved it, you should pick it up very fast.

In the future I hope to generate a 1:1 import tool that will download, convert and install packages for you automatically.

Cheers guys!

Embedded boards, finally!

December 19, 2016 Leave a comment

I was about to conclude that this day horizontally sucked beyond measure, but just as I thought as much -the door bell rang. It was FedEx (ta da!) with not one package, but two packages of super nerdy goodness. And here I was sure these puppies wouldnt arrive until after Xmas!

Top the x86 UP board, left bottom a Raspberry PI 3, bottom right the ODroid XU4

Top the x86 UP board, left bottom a Raspberry PI 3, bottom right the ODroid XU4

The ODroid XU4

A while back I ordered the very exciting “Raspberry PI 3 killer” ODroid XU4. It’s a bit of a unicorn, said to be roughly 10 times faster than the Raspberry PI 3. Honestly, having looked at the specs I can’t imagine it being more than 3 to 4 times faster; depending greatly on what your application is doing and the operative system in question. Here is the full spec sheet:

  • Samsung Exynos 5422 Cortex™-A15 2 GHz and Cortex™-A7 Octa core CPUs
  • Mali-T628 MP6(OpenGL ES 3.0/2.0/1.1 and OpenCL 1.1 Full profile)
  • 2 Gbyte LPDDR3 RAM PoP stacked
  • eMMC 5.0 HS400 Flash Storage
  • 2 x USB 3.0 Host, 1 x USB 2.0 Host
  • Gigabit Ethernet port
  • HDMI 1.4a for display

As you can see the ODroid comes armed with 8 CPU cores while the Raspberry PI 3 (RPI3) has only 4. It also comes with twice the RAM (which really impacts performance), and tests have shown the ODroid disk/IO speed is roughly double that of the RPI3. But the cores is what caught my eye, because 8 cores means 8 simultaneous threads. This means code written for node.js, apache [php] or indeed custom, natively compiled Free Pascal servers will be able to handle twice the payload straight off the bat. For stateless protocols like http, I am guessing a performance factor of 3 to 1 compared to an RPI3.

Having said all this, there will be exceptions where the PI3 performs equal of better. The RPI3 SoC have better HD video functionality, so the ODroid have to work harder to produce the same.

For those of you thinking the ODroid will solve all your Amiga emulation problems, the answer is yes. It is significantly faster than the RPI3. But never forget that single threaded applications like UAE (Unix Amiga Emulator) involves a high degree of chipset synchronization. If the chipset performs out of sync (for instance if the blitter finishes faster than it should), all manner of problems will occur. So all that synchronization causes some parts to wait. Meaning no matter how fast your computer is (even my Intel i7 CPU) UAE will never reach peak performance. It will never use the full capacity of a single core due to synchronization.

A small note is also popularity, which means less updates. ODroid has a somewhat slower update cycle than Raspberry. It has thousands of users, but it’s not even close to getting the attention of the Raspberry PI project. And where the Raspberry website has been community oriented, with inspiring graphics, free tutorials and a huge forum from day one – the ODroid has nothing that even compares.

From what I read and hear the biggest problem has been kernel updates. But, seriously, that is often a Linux fetish. Unless it’s a super important update, like going from 32 to 64 bit or patching a really terrible security flaw – most users are not going to be bothered by a 2 versions old kernel. You still have access to the proverbial library of Alexandria that is aperture package manager (apt-get command) and compiling a few programs from code is not that hard. It’s typically download, unpack, configure, make and install – and that’s it.

Naturally, considering the faster CPU of the ODroid, double the ram, double the IO speed – emulators like UAE will be awesome. ODroid is also the only ARM SoC out there in this price range that plays Sega Saturn and PSX 2 games without any problems. And it will also be far better suited for servers, be it natively compiled freepascal servers, mono ASP.net or Smart Pascal [node.js] work.

The UP x86 board

The second package contained another embedded board, this time the x86 based UP board. I bought the most expensive model they had, containing 4 gigabytes of ram and 64 GB EMMC on-board storage. The board sports a 64 bit Intel® Atom™ x5 Z8350 processor, running as high as 1.92 GHz. Between Raspberry PI 3, ODroid XU4 and UP – let there be no doubt which model will come out on top.

  • Intel® Atom™ x5 Z8350 Processor 64 bit – up to 1.92GHz
  • Intel® HD 400 Graphics ,12 EU GEN 8, up to 500MHz Support DX*11.1/12, Open GL*4.2, Open CL*1.2 OGL ES3.0, H.264, HEVC(decode), VP8
  • 4GB DDR3L
  • 64GB eMMC
  • 4 x USB2.0 external connector
  • 2 x USB2.0 port (pin header)
  • 1 x USB 3.0 port
  • 1 x Gb Ethernet (full speed) RJ-45
  • HDMI (DSI / eDP)
  • MIPI-CSI Camera interface
  • 5V DC-in @ 3A 5.5/2.1mm jack

Where the ODroid is rumoured to be 10 times faster than a RPI3, that is a statement closer to an urban myth rather than fact; The UP board on the other hand IS without a shadow of a doubt 10 times faster (and then some), no question about it. Since this is essentially a vanilla x86 SoC PC, the world is your oyster. The full onslaught of x86 software is at your disposal, be it Windows or Linux you chose as your base.

x86 UP board, same size as the PI3 but packing a hell of a punch!

x86 UP board, same size as the PI3 but packing a hell of a punch!

The board has no problems running Windows 10, Ubuntu (full 64 bit version) and Android. And it’s actually more responsive than a few laptops on sale. I wanted a cheap laptop for dedicated Amiga Emulation – but having tested both the low-end Asus and Dell models it just left me wanting. The problem with cheap model laptops is often the absurd memory they ship with (1 to 2 gigabyte). The devices spend more time swapping data back and forth between ram and disk than they do running your code!

This is not the case for the UP board thanks to the on-board 4 gigabytes of ram.

Being a Embarcadero Delphi and Smart Mobile Studio (object pascal for JavaScript) developer this board is perfect for my needs. It has so much more to offer than the cheap ARM boards and is something I can use to create custom hardware projects and avoid many of the adaptation problems associated with ARM Linux.

While I love Raspberry PI 3, Linux takes some getting used to. There are things that takes me days to figure out on Linux that I completed in minutes on Windows (I have been using Windows since the beginning after all). This will change as my skill and insight into Linux matures, but if I can choose between the RPI3 and the UP board, I would pick the UP board every time.

Price is a factor here. RPI3 sells for between $35-40, the ODroid retails for $99 while the x86 UP board can be yours for $150. But you can also buy a cheaper model with less ram and EMMC storage. The UP provider has the following options:

  • $99 – 2 gigabyte memory, 16 gigabyte EMMC storage
  • $109 – 2 gigabyte memory, 32 gigabyte EMMC storage
  • $129 – 4 gigabyte memory, 32 gigabyte EMMC storage
  • $150 – 4 gigabyte memory, 64 gigabyte EMMC storage

If you’re thinking “oh, for that price I could get 2, 3 and 4 PI’s respectively!”, keep in mind the level of CPU power, graphics speed and available software here. The fact that you can install and run Windows 10 and just copy over your Delphi or Smart applications is really sweet.

And if you are into emulation then naturally, Windows has the latest and greatest. Things like EmulationStation is going to run so much better on this device than anything else at this price out there, especially if you get the x86 Linux edition. You can run the latest WinUAE on Windows, giving you full PPC emulation and essentially booting straight into Workbench and OS4.1. $150 for an OS4.x capable SoC that is x86 based makes a hell of a lot more sense than forking out $489 for the A1222 low-end PPC based Amiga NG board planned for 2017. I mean, who the hell buys PPC in 2017 (!) Emulation will always be a little bit slower than the real thing, but we are talking negligible.

And with the UP board you can also re-cycle the hardware when you get bored with OS 4. I mean, there are only so many things you can do with a modern Amiga. It is great fun for enthusiasts like myself, but I would much rather run a juiced up version of OS 3.9 with my massive collection of WHDLoad software, cd32 software and modern compilers (like Free Pascal 3.1 which work brilliantly on an emulated, classic Amiga).

Programming

For the developer the UP board gives you the same choices you enjoy on your Windows development machine in general. You can use it to deliver your Delphi, C++ builder or Smart Pascal solutions. If you happen to own a Microsoft Embedded license its performance will be greatly enhanced since you can drop a lot of the “standard” stuff that has to ship with Windows. Remember, a standard Windows  installation is written to work on millions of PC’s and equal number of different hardware configurations. For customized, single purpose applications (like a kiosk system, information booth, cash machine type system) you will be able to cut out quite a lot. Do you need support for printers? Do you need driver-collection for hardware that is not there? Windows embedded allows you to cut the disk image down to the bones, and it’s also written to run on slower CPU’s than what people in general own – so the performance is much better.

  • Run your Delphi projects, it’s a normal PC after all
  • Run your node.js Smart Pascal projects with ease
  • Make use of nodewebkit for Smart Pascal to create full screen, desktop oriented software, enjoy the full scope of GPU powered CSS
  • Enjoy the debugging tools you already know
  • Run familiar databases like MSSQL, MySQL and Firebird with a more friendly and developed editors and tools
  • Use the more friendly backup solution that ships with Windows rather than some cryptic Linux solution (although some Linux versions that have desktop control-panels are just as great!)
  • Use GPIO ports from Delphi and C++ builder (!)
  • Just hook your UP board into your network and install/setup via remote desktop. It saves a lot of time.

If you are a Delphi programmer looking for a reasonable embedded board to ship your killer Windows-based product, the UP board is by far the best option I have seen (and I have tested quite a few board out there!).

The problem with high-end boards is not just the initial price, which can be anything from $300 to $400. Sure, these boards Intel i2 or i3 processors (much faster), but you end up paying extra for everything. Need a new ram module? That will set you back a pretty penny! Want GPIO? Again we are talking $100+ just to get that.

By the time you sum up the expenses, you realize it would have been cheaper to just visit the local computer store and bought a mico-atx board with more ram and a faster processor (!). Micro-atx is not that big, perhaps 2 times larger than the UP board (if you place them into a square). The micro-atx is often to high to be practical embedded boards where you want to cram as much hardware as you can into something the size of a router or set-top-box. The heat sink hovers over the motherboard like the eye of london.

Here is what you should have in mind:

  • Is size a factor?
    • No
      • Buy a cheap mico-atx pc
    • Yes
      • Take a look at the boards listed below
  • Do you need Windows support?
    • No
      • Get a ARM based device
    • Yes
  • Do you need an i3 or i5 CPU?
  • Do you need GPIO built-in?

Note: The UP project is presently busy working on their second kickstarter, which is cleverly called “Up 2” (sigh). This board is slightly larger, being dubbed “UP squared”, but there is a good reason for that. First of all it ships with the more powerful Intel® Pentium™ N4200 2.5 GHz CPU, up to 8 gigabyte of memory and 128 gigabyte emmc storage. Just like the present UP board you will be able to pick a configuration that matches your wallet, but they are aiming at roughly the same price-range as they have now. Head over to the UP project and have a peek at the specs!

Negatives

So far the only negative thing about the UP board is the speed of the emmc storage. As you probably know, emmc is a storage medium designed to be a cheap alternative to SSD. But when it comes to speed it can be anything from SD card to USB 3 in actual performance. This is very vendor spesific and obviously the cheaper models are going to be the slowest. So the first thing you are going to notice is that even though this is a PC, installing things takes a lot longer.

You can however enter the bios and boot from a USB 3 stick. For homebrew projects that shouldnt matter much, and these days a sexy and small 128 or 256 gigabyte (you know, those tiny usb storage devices that is barely the USB socket and little else) is affordable.

I find myself having to look for negatives here. I do think the UP organization should do something about the startup of the device. When you boot the UP logo is displayed, but they should have added a line of text about what key to press for the bios. I ended up going through the typical ones, F1, F2, F8 and F10 which did nothing. The next key was printscreen, before i finally hit DEL and the bios editor came up.

Insignificant, but a small detail that would make their product more polished.

A far worse notion is how they charge money for branding. When the product boots the UP logo comes into view for a couple of seconds (in full-screen). If you want to replace that, the minimal fee is $500 (for a small picture). This is something that simply infuriates me, because you cant change it.

When you buy an embedded board for production purposes, hidden costs such as this for something as simple as a picture – is completely unacceptable. I sincerly hope they drop this practise, because I will not use a board with a fixed boot picture in the embedded systems I deliver. There are plenty of other boards about there with similar specs at competitive prices. Being able to brand your own system is considered normal in 2016 and it has been common for years now.

At least an option in the bios for removing or hiding the UP logo should be in place.

 

The test

For the next few days before and after Xmas, I’ll be playing with these boards as much as time allows. I have already installed the latest Ubuntu on the UP board – and it performed brilliantly. I am presently giving Windows 10 a test drive, and my primary aim will be Smart Mobile Studio graphics demos and UAE running Amiga OS 4.1 final edition. I will also test how emulators work. This is especially exciting for the ODroid since that is the one most people will pick as an alternative to Raspberry PI 3.

If you are a dedicated retro-gamer or just love all things Amiga then again, the UP board should be of special interest to you. It will set you back $150, but considering that it has the exact same form-factor as the Raspberry PI (except its components go a few mm. higher) is considerably faster (in a league way beyond both the PI and ODroid) – this could be your ticket to get a cheap “next-gen” emulated Amiga. It will run OS 4.1 final without problems under WinUAE and it will be a pleasant experience.

I will give you updates as the frame rates, execution speed and overall performance comes in. Oh and the tests will obviously use the RPI3 as the baseline.

Cheers guys!

 

Ghost of xmas past, or better known as the folder where old projects and abandoned ideas end up

December 8, 2016 Leave a comment

I think everyone has a folder where they stuff old units, test projects and well – all those ideas that seemed awesome at the time, but you either discovered that it was crap, not enough time, or just forgot all about it. It happens. What can I say.

Yeah, I have one of those as well. And I also have those duplicate folders. You know, that time when you decided to make backups for everything – but then you got a new PC and now you dont have a clue which version to keep? And when you finally sit down with beyondcompare to get this sorted, you have managed to add to both copies.

Well, needless to say I had a day I just went apeshit about a year ago and sorted everything. And whatever did not fit into my neat new code-folder(s) was mercilessly stuffed into the dark, obscure folder known only as “stuff” or “misc”.

My first basic compiler, awwwww how cute

Well its not all rubbish! I did find a special version of my kitchen-sink research IDE, the IDE I use when trying out new code for DWScript, PaxCompiler and FreePascal. I think it has 4 compilers in it or something, yet its under 30 megabytes in size! Pretty neat 🙂

Visual Basic to Smart Pascal, awww.... just want to snuggle it :)

Visual Basic to Smart Pascal, awww…. just want to snuggle it 🙂

It also features a unified file-system! You can mount packages, FTP folders, network paths or local folders – the IDE could not care less, it is completely abstracted from the actual filesystem and relates only to symbols and identifiers. The actual storage is dealt with by the filesource classes.

Package, ftp, socket, local folder - the IDE could not care less

Package, ftp, socket, local folder – the IDE could not care less

The Basic dialect here is essentially classical Visual Basic. The way you would write classes and deal with instances before dot net came along and forced all languages to look the same. I kinda like old visual basic, it had soul. It was completely useless except for scripting, and Delphi made it look like a joke – but as far as basic dialects go, it’s probably one of the cleanest and easiest to read.

DWScript, QTX Pascal (a custom fork of DWScript with a lot of cool features, some that Eric has added afterwards), Basic - and even freepascal! Thats the mother of all kitchen sinks!

DWScript, QTX Pascal (a custom fork of DWScript with a lot of cool features, some that Eric has added afterwards), Basic – and even freepascal! Thats the mother of all kitchen sinks!

The loss of QTX Pascal is a bit sad. I spent a couple of months doing my own fork of DWScript. Most of the features I added have now been included (although in another form) by Eric. But the codegen produced faster javascript. One of the things I spent some time on was optimization. Like getting rid of “variable = 0” type initialization if an assignment followed. I also added BeforeDestruction() and AfterConstruction() calls. This made an RTL a lot easier to write, but also more overhead. I was about to do conditional testing (so these would only be called if you actually used them) when I had to stop and work on Smart Mobile Studio again.

N++, exotic and functional!

This was one of my favorite language research projects. And dare I say, it is way ahead of it’s time. The idea is simple: with the future of computing being distributed, cloud based and powered by a multitude of computing modules from the US to South Africa — what would a programming language look like if built for 100% asyncronous, distributed execution?

Consider for instance, the execution of a procedure. Simple right? Well, in a truly distributed system – that procedure could execute anywhere. The whole point here is to utilize the combined power of each module (pc); which means a tasks could execute on different computers. A procedure could also be split up by the runtime environment and once again, be executed all over the place – with the dispatcher keeping track of when it’s done and whatever the result was (if any).

Only a spesific and generated on-demand “runtime context” would follow the task. This would contain things like variables it needs to execute, public symbols, basicaly just enough for the code to run successfully. This also includes resource forks (a term i used for accessing resources regardless of where they may be on the network).

The language design was actually done using Smart Mobile Studio (you can read more about the initial ideas here), and the first test modules ran on node.js (so no fancy graphics available sadly).

But it was incredibly fun to play with! But also time consuming to design a language for an execution model that havent really been invented yet. You can read more about some of my ideas for the execution model here.

I dont even think this execution model is out of the MIT labs yet — but it will once quantum compute models become commercially available (click here for a Berkley University published introduction to the basic computational models used in quantum information theory).

An N++ procedure (actually a microservice that consumed a WSDL service and calls it) looks like this:

program("service_test") {

  handshake {

    input {
      /* Consume WSDL Web-Service Endpoint */
      service1 @ service[URI:"http://www.test.com/SOAP/myService/WSDL/",
                 serviceType:SoapService];
    }

    output {
        myProcess @ process("self");
        stdio @ pipe("stdout");
      }
  }

  /* Execute RPC call */
  execute (stdio,service1)  {
    stdio:writeln("Calling webservice");
    execute (*)  {
      var int32 result = 0;
      set result = service1:getUserId("quartex","secret");
      stdio:writelnF("UserID on server={0}", result);
    } fail (e) {
      stdio.writelnF("Calling soap service failed: {0}",e);
      proceed;
    }
  }

  /* Exit code for process */
  set myProcess:exitCode = 0;
}

RML, Reduced markup language parser and runtime

Dont you just hate it when you lose the source-file for a programming language you made in a day, so now you dont have a clue what the syntax really was? I know, I know, but I honestly dont have the syntax or examples. It does with a disk ages ago.

RML was written to simplify generating HTML documents. There are a few things that you can optimize when it comes to HTML. First of all you know that the browser will read the document from top and downwards. This might not mean much at first glance, but it actually means that a language doesnt need the abillity to call emitter-code that has already executed (above). Think of it as being able to omit calls to procedure behind you in the source. That would be a mess for a language like delphi – but it doesnt impact a language that execute linear without any forms of jumps!

I seem to remember it looked something like this:

createobject(TRMLObject)
{
   writestr("testing");
   if(a=12)
   {
       readstr("mer testing");
   }
}
createobject(TRMLHeading)
{
   settitle("testing second object");
   if(a=12)
   {
       writestr("heading object successfully done");
   }
}
createobject(TRMLObject)
{
   writestr("testing third object");
   if(a=12)
   {
       readstr("mer testing from da third object");
   }
}

The idea is ofcourse that you have some pre-defined objects, a bit like custom-controls in Smart Pascal, and you create and populate these in a top-down fashion.
Once defined, generating quite advanced HTML documents can be done extremely fast. Much faster than ASP, DWS or php since no jumps are involved.

Here is how you compile and run a small debug session (drop a memo and button on a form first):

  var
    FContext: TRMLParseContext;
    FText:  String;
  begin
    FText:= 'createobject(TRMLObject)'
    +       '{'
    +       '   writestr("testing");'
    +       '   if(a=12)'
    +       '   {'
    +       '       readstr("mer testing");'
    +       '   }'
    +       '}'
    +       'createobject(TRMLHeading)'
    +       '{'
    +       '   settitle("testing second object");'
    +       '   if(a=12)'
    +       '   {'
    +       '       writestr("heading object successfully done");'
    +       '   }'
    +       '}'
    +       'createobject(TRMLObject)'
    +       '{'
    +       '   writestr("testing third object");'
    +       '   if(a=12)'
    +       '   {'
    +       '       readstr("mer testing from da third object");'
    +       '   }'
    +       '}';

    // double the code, just to get some payload
    FText := FText + FText;

    memo1.text:='';

    If RMLAllocContext(FContext,FText,[TRMLObject,TRMLHeading]) then
    Begin
      try
        if RMLCompile(FContext) then
        Begin
          caption:='Compilation was a success';
          If RMLExecute(FContext) then
          Begin
            Caption:='Executed successfully';
            memo1.text:=FContext.pcOutput;
          end;
        end else
        caption:=Format('compilation failed [%s]',[FContext.pcError]);
      finally
        RMLDisposeContext(FContext);
      end;
    end;

And here is the full code for the parser and runtime (still a few tidbits to work on, but its childs play). If you can make sense of it, knock yourself out 🙂

  unit rml;

  interface

  uses sysutils, classes, dialogs;

  type

  TRMLObject  = Class;

  TRMLClass     = Class of TRMLObject;
  TRMLOperator  = (opLess,opMore,opEquals,opNotEquals);
  TRMLDataType  = (daString,daNumber,daBoolean,daCustom);
  TRMLEntryType = (etAssignment,etMethod,etRepeat,etCreate,etIf);
  TRMLObjProc   = Function (Const EntryData:Pointer):Boolean of Object;

  (* represents a single condition in an IF statement *)
  PRMLCondition = ^TRMLCondition;
  TRMLCondition = Packed Record
    coSource:   String;
    coTarget:   String;
    coOperator: TRMLOperator;
  End;

  (* represents a parsed code declaration *)
  PRMLEntryDeclaration = ^TRMLEntryDeclaration;
  TRMLEntryDeclaration = Packed Record
    cxToken:      String;
    cxCondition:  String;
  End;

  (* represents a compiled parameter *)
  PRMLParameter = ^TRMLParameter;
  TRMLParameter = Packed Record
    prType:   TRMLDataType;
    prValue:  Pointer;
    prSize:   Integer;
  End;

  (* represents a compiled code entry *)
  PRMLEntryData = ^TRMLEntryData;
  TRMLEntryData = Packed Record
    edtype:         TRMLEntryType;
    edDeclaration:  TRMLEntryDeclaration;
    edObject:       TRMLObject;
    edParent:       PRMLEntryData;
    edMethod:       TRMLObjProc;
    edContext:      Pointer;
    edConditions:   Array of PRMLCondition;
    edParameters:   Array of PRMLParameter;
    edSubEntries:   Array of PRMLEntryData;
  End;

  PRMLParseContext = ^TRMLParseContext;
  TRMLParseContext = Packed Record
    pcSignature:  Integer;
    pcCol:        Integer;
    pcRow:        Integer;
    pcPos:        Integer;
    pcLen:        Integer;
    pcData:       String;
    pcError:      String;
    pcOutput:     String;
    pcRoot:       TRMLEntryData;
    pcClasses:    Array of TRMLClass;
  End;

  TRMLReadProc  = Function  (Var OutData;var Bytes:Integer):Boolean of Object;
  TRMLWriteProc = Function  (Var InData;Const Bytes:Integer):Boolean of Object;
  TRMLProcEntry = Function  (Const Entry:PRMLEntryData):Boolean of Object;

  PRMLObjectIndex = ^TRMLObjectIndex;
  TRMLObjectIndex = Packed Record
    oiMethods:    Array of record
                    omName:   String;
                    omSyntax: Array of TRMLDataType;
                    omEntry:  TRMLProcEntry;
                  end;
    oiProperties: Array of record
                    oiName:   String;
                    oiRead:   TRMLReadProc;
                    oiWrite:  TRMLWriteProc;
                    oiType:   TRMLDataType;
                  end;
  End;

  TRMLObject = Class(TObject)
  Private
    FIndexData: TRMLObjectIndex;
    FIndexPTR:  PRMLObjectIndex;
  Private
    Function    DoWriteStr(Const Entry:PRMLEntryData):Boolean;
    Function    DoReadStr(Const Entry:PRMLEntryData):Boolean;
  protected
    Procedure   Output(Const Context:PRMLParseContext;Const Value:String);
    Procedure   RegisterProperty(Const Name:String;Const DataType:TRMLDataType;
                Const _Read:TRMLReadProc;Const _Write:TRMLWriteProc);
    Procedure   RegisterMethod(Const Name:String;
                Const Syntax: Array of TRMLDataType;
                Const Entry: TRMLProcEntry);
  Public
    Property    ObjectIndex:PRMLObjectIndex read FIndexPTR;
    Constructor Create;virtual;
  End;

  TRMLHeading = Class(TRMLObject)
  Private
    FTitle:     String;
    Function    DoSetTitle(Const Entry:PRMLEntryData):Boolean;
    Function    DoReadTitle(Var OutData;var Bytes:Integer):Boolean;
    Function    DoWriteTitle(Var InData;Const Bytes:Integer):Boolean;
  Public
    property    Title:String read FTitle write FTitle;
    Constructor Create;override;
  End;

  Function  RMLAllocContext(var Context:TRMLParseContext;
            Const Source:String;Const ClsBase:Array of TRMLClass):Boolean;
  Function  RMLDisposeContext(Var Context:TRMLParseContext):Boolean;
  Function  RMLCompile(Var Context:TRMLParseContext):Boolean;
  Function  RMLExecute(Const Context:TRMLParseContext):Boolean;
  Function  RMLParseEntry(Value:String;
            var Declaration:TRMLEntryDeclaration;var Error:String):Boolean;

  implementation

  //###########################################################################
  // TRMLHeading
  //###########################################################################

  Constructor TRMLHeading.Create;
  Begin
    inherited;
    RegisterMethod('settitle',[daString],DoSetTitle);
    RegisterProperty('title',daString,DoReadTitle,DoWriteTitle);
  end;

  Function TRMLHeading.DoSetTitle(Const Entry:PRMLEntryData):Boolean;
  var
    FTemp:  String;
  Begin
    result:=length(Entry^.edParameters)>0;
    If result and (Entry^.edParameters[0].prType=daString) then
    DoWriteTitle(Entry^.edParameters[0].prValue^,
    Entry^.edParameters[0].prSize);
  end;

  Function TRMLHeading.DoReadTitle(Var OutData;var Bytes:Integer):Boolean;
  Begin
    Bytes:=Length(FTitle);
    If Bytes>0 then
    move(FTitle[1],outData,Bytes);
  end;

  Function TRMLHeading.DoWriteTitle(Var InData;Const Bytes:Integer):Boolean;
  Begin
    SetLength(FTitle,bytes);
    if Bytes>0 then
    move(inData,FTitle[1],Bytes);
  end;

  //###########################################################################
  // TRMLObject
  //###########################################################################

  Constructor TRMLObject.Create;
  begin
    inherited;
    FIndexPTR:=@FIndexData;
    RegisterMethod('writestr',[daString],DoWriteStr);
    RegisterMethod('readstr',[daString],DoReadStr);
  end;

  Procedure TRMLObject.Output(Const Context:PRMLParseContext;
            Const Value:String);
  Begin
    Context^.pcOutput:=Context^.pcOutput + Value;
  end;

  Function TRMLObject.DoReadStr(Const Entry:PRMLEntryData):Boolean;
  var
    FText:  String;
  Begin
    result:=True;
    With Entry^ do
    Begin
      FText:='Token: ' + edDeclaration.cxToken + #13#10;
      FText:=FText + 'Condition: ' + edDeclaration.cxCondition + #13#10;
      FText:=FText + #13#10;
      Output(edContext,FText);
    end;
  end;

  Function TRMLObject.DoWriteStr(Const Entry:PRMLEntryData):Boolean;
  var
    FText:  String;
  Begin
    result:=True;
    With Entry^ do
    Begin
      FText:='Token: ' + edDeclaration.cxToken + #13#10;
      FText:=FText + 'Condition: ' + edDeclaration.cxCondition + #13#10;
      FText:=FText + #13#10;
      Output(edContext,FText);
    end;
  end;

  Procedure   TRMLObject.RegisterProperty(Const Name:String;
              Const DataType:TRMLDataType;
              Const _Read:TRMLReadProc;Const _Write:TRMLWriteProc);
  var
    FCount: Integer;
  Begin
    //FCount:=high(FIndexData.oiProperties) - Low(FIndexData.oiProperties) + 1;
    FCount:=Length(FIndexData.oiProperties);
    SetLength(FIndexData.oiProperties,FCount+1);
    FIndexData.oiProperties[FCount].oiName:=Name;
    FIndexData.oiProperties[FCount].oiRead:=_Read;
    FIndexData.oiProperties[FCount].oiWrite:=_Write;
    FIndexData.oiProperties[FCount].oiType:=DataType;
  end;

  Procedure   TRMLObject.RegisterMethod(Const Name:String;
              Const Syntax: Array of TRMLDataType;
              Const Entry: TRMLProcEntry);
  var
    FCount: Integer;
    FTemp:  Integer;
  Begin
    //FCount:=high(FIndexData.oiMethods) - Low(FIndexData.oiMethods) + 1;
    FCount:=Length(FIndexData.oiMethods);
    SetLength(FIndexData.oiMethods,FCount+1);
    FIndexData.oiMethods[FCount].omName:=Name;
    FIndexData.oiMethods[FCount].omEntry:=Entry;

    //FTemp:=high(Syntax) - Low(Syntax) + 1;
    FTemp:=Length(Syntax);
    If FTemp>0 then
    Begin
      SetLength(FIndexData.oiMethods[FCount].omSyntax,FTemp);
      for FTemp:=low(syntax) to high(syntax) do
      FIndexData.oiMethods[FCount].omSyntax[FTemp]:=Syntax[FTemp];
    end;
  end;

  //###########################################################################
  // RML util methods
  //###########################################################################

  Function RMLContainChars(Const Value:String;const Chars:String):Boolean;
  var
    x:  Integer;
  Begin
    result:=True;
    for x:=1 to length(chars) do
    Begin
      if pos(chars[x],Value)<1 then
      Begin
        result:=False;
        Break;
      end;
    end;
  end;

  Function  RMLScanFor(const Value:String;Const Target:CHAR;
            Const Breakers:String;var Len:Integer):Boolean;
  var
    xpos: Integer;
  Begin
    result:=False;
    Len:=-1;
    xpos:=1;
    while xpos<=Length(Value) do     Begin       If Value[xpos]=Target then       Begin         Len:=xpos-1;         Result:=True;         Break;       end else       Begin         if pos(Value[xpos],Breakers)>0 then
        Break;
      end;
      inc(xpos);
    end;
  end;

  Function RMLisNumber(Const Value:String):Boolean;
  const
    CHARSET = '0123456789';
  var
    x:  Integer;
  Begin
    Result:=True;
    for x:=1 to length(Value) do
    Begin
      if pos(Value[x],CHARSET)<1 then
      Begin
        result:=False;
        Break;
      end;
    end;
  end;

  Function RMLIsBoolean(Const Value:String):Boolean;
  var
    FTemp:  String;
  Begin
    FTemp:=lowercase(trim(Value));
    result:=(FTemp='false') or (FTemp='true');
  end;

  Function RMLisString(Const Value:String):Boolean;
  var
    x:      Integer;
    FLeft:  Integer;
  Begin
    result:=False;
    FLeft:=0;
    (* check left side *)
    for x:=1 to length(Value) do
    Begin
      if Value[x]='"' then
      Begin
        FLeft:=x;
        Result:=True;
        Break;
      end else
      if Value[x]<>#32 then
      Break;
    end;
    (* check right side *)
    If result then
    Begin
      for x:=Length(Value) downto 1 do
      Begin
        if Value[x]='"' then
        Begin
          If x>FLeft then
          Break else
          Begin
            Result:=False;
            Break;
          end;
        end else
        if Value[x]<>#32 then
        Break;
      end;
    end;
  end;

  Function  RMLParseEntry(Value:String;
            var Declaration:TRMLEntryDeclaration;
            var Error:String):Boolean;
  var
    xpos: Integer;
  Begin
    fillchar(Declaration,SizeOf(Declaration),0);
    Result:=RMLContainChars(value,'()');
    if Result then
    Begin
      Result:=RMLScanFor(value,'(',')',xpos);
      if result then
      Begin
        Declaration.cxToken:=trim(copy(value,1,xpos));
        delete(value,1,xpos+1);
        Result:=RMLScanFor(value,')','(',xpos);
        if result then
        Begin
          Value:=TrimRight(Value);
          Result:=xpos=(length(value)-1);
          if result then
          Declaration.cxCondition:=trim(Copy(Value,1,xpos));
        end;
      end;
    end;
    If not Result then
    Error:='Invalid entry <' + value + '>';
  end;

  Function  RMLAllocContext(var Context:TRMLParseContext;
            Const Source:String;Const ClsBase:Array of TRMLClass):Boolean;
  var
    FCount: Integer;
  Begin
    If Context.pcSignature=SizeOf(Context) then
    RMLDisposeContext(Context);

    fillchar(Context,SizeOf(Context),#0);
    Context.pcSignature:=SizeOf(Context);
    Context.pcLen:=Length(Source);
    Context.pcData:=Source;

    FCount:=High(clsBase) - low(clsBase)+1;
    If FCount>0 then
    Begin
      SetLength(Context.pcClasses,FCount);
      for FCount:=Low(clsBase) to high(clsBase) do
      Context.pcClasses[FCount]:=clsBase[FCount];
    end;

    result:=True;
  end;

  Procedure RMLDisposeEntryData(Const Data:PRMLEntryData);
  var
    FTemp:  Integer;
    x:      Integer;
  Begin
    (* dispose of condition data *)
    FTemp:=length(Data^.edConditions);
    While FTemp>0 do
    Begin
      Dispose(Data^.edConditions[FTemp-1]);
      dec(FTemp);
    end;
    SetLength(Data^.edConditions,0);

    (* dispose of parameter data *)
    FTemp:=length(Data^.edParameters);
    While FTemp>0 do
    Begin
      If length(Data^.edParameters)>0 then
      Begin
        for x:=Low(Data^.edParameters) to high(Data^.edParameters) do
        If Data^.edParameters[x]^.prSize>0 then
        FreeMem(Data^.edParameters[x]^.prValue);
      end;
      Dispose(Data^.edParameters[FTemp-1]);
      dec(FTemp);
    end;
    SetLength(Data^.edParameters,0);

    (* dispose of sub entries *)
    //Ftemp:=High(Data^.edSubEntries)-Low(Data^.edSubEntries)+1;
    FTemp:=Length(Data^.edSubEntries);
    While FTemp>0 do
    Begin
      RMLDisposeEntryData(Data^.edSubEntries[FTemp-1]);
      Dec(FTemp);
    end;
    SetLength(Data^.edSubEntries,0);

    (* dispose of script object *)
    If  not (Data^.edtype in [etIf,etRepeat,etMethod])
    and (Data^.edObject<>NIL) then
    Data^.edObject.free;

    (* dispose of entry *)
    Dispose(Data);
  end;

  Function  RMLDisposeContext(Var Context:TRMLParseContext):Boolean;
  var
    FCount: Integer;
  Begin
    Result:=Context.pcSignature=SizeOf(Context);
    If Result then
    Begin
      Context.pcSignature:=0;
      Context.pcData:='';
      Context.pcError:='';
      FCount:=Length(Context.pcRoot.edSubEntries);
      //FCount:=High(Context.pcRoot.edSubEntries)
      //- Low(Context.pcRoot.edSubEntries) + 1;
      While FCount>0 do
      Begin
        RMLDisposeEntryData(Context.pcRoot.edSubEntries[FCount-1]);
        dec(FCount);
      end;
      SetLength(Context.pcRoot.edSubEntries,0);
    end;
  end;

  //###########################################################################
  // RML core methods
  //###########################################################################

  Function  RMLImplements(Const MethodName:String;
            Const obj:TRMLObject):Boolean;
  var
    FTable: PRMLObjectIndex;
    FCount: Integer;
  Begin
    Result:=Obj<>NIL;
    If result then
    Begin
      (* get object inex *)
      FTable:=Obj.ObjectIndex;
      //FCount:=High(FTable^.oiMethods) - low(FTable^.oiMethods) + 1;
      FCount:=Length(FTable^.oiMethods);
      Result:=FCount>0;
      If Result then
      Begin
        for FCount:=low(FTable^.oiMethods) to high(FTable^.oiMethods) do
        Begin
          Result:=FTable^.oiMethods[FCount].omName=MethodName;
          If Result then
          Break;
        end;
      end;
    end;
  end;

  Function  RMLGetMethodEntry(Const MethodName:String;
            Const obj:TRMLObject;var outEntry:TRMLObjProc):Boolean;
  var
    FTable: PRMLObjectIndex;
    FCount: Integer;
  Begin
    Result:=Obj<>NIL;
    If result then
    Begin
      (* get object inex *)
      FTable:=Obj.ObjectIndex;
      //FCount:=High(FTable^.oiMethods) - low(FTable^.oiMethods) + 1;
      FCount:=Length(FTable^.oiMethods);
      Result:=FCount>0;
      If Result then
      Begin
        for FCount:=low(FTable^.oiMethods) to high(FTable^.oiMethods) do
        Begin
          Result:=FTable^.oiMethods[FCount].omName=MethodName;
          If Result then
          Begin
            outEntry:=TRMLObjProc(FTable^.oiMethods[FCount].omEntry);
            Break;
          end;
        end;
      end;
    end;
  end;

  Function  RMLCreateObject(var Context:TRMLParseContext;
            Const Objname:String;var outObject:TRMLObject;
            var Error:String):Boolean;
  var
    FCount: Integer;
  Begin
    FCount:=High(Context.pcClasses) - Low(Context.pcClasses) + 1;
    Result:=FCount>0;
    if Result then
    Begin
      For FCount:=Low(Context.pcClasses) to high(Context.pcClasses) do
      Begin
        Result:=lowercase(Context.pcClasses[FCount].ClassName)=lowercase(objName);
        If result then
        Begin
          outObject:=Context.pcClasses[FCount].Create;
          Break;
        end;
      end;
    end;
    If not Result then
    Error:='Unknown class <' + Objname + '>';
  end;

  Function  RMLAddEntry(var Context:TRMLParseContext;
            Var Declaration:TRMLEntryDeclaration;
            Root:PRMLEntryData;var NewEntry:PRMLEntryData;
            var Error:String):Boolean;
  var
    FCount: Integer;
    x:      Integer;
    FTemp:  String;
    FLen:   Integer;
    FPar:   PRMLParameter;
  Begin
    Result:=Root<>NIL;
    If result then
    Begin
      (* create new entry *)
      new(NewEntry);

      (* Reset entry record *)
      NewEntry^.edType:=etAssignment;
      NewEntry^.edObject:=NIL;
      NewEntry^.edMethod:=NIL;
      SetLength(NewEntry^.edConditions,0);
      SetLength(NewEntry^.edParameters,0);
      SetLength(NewEntry^.edSubEntries,0);

      (* Set basic values *)
      NewEntry^.edParent:=Root;
      NewEntry^.edContext:=@Context;
      newEntry^.edDeclaration:=Declaration;

      (* insert entry into parent *)
      FCount:=Length(Root^.edSubEntries);
      SetLength(Root^.edSubEntries,FCount+1);
      Root^.edSubEntries[FCount]:=NewEntry;

      (* tokenize *)
      If declaration.cxToken='createobject' then
      Begin
        NewEntry^.edtype:=etCreate;
        Result:=RMLCreateObject
          (
          Context,declaration.cxCondition,
          NewEntry^.edObject,Error
          );
      end else

      if declaration.cxToken='if' then
      Begin
        NewEntry^.edtype:=etIF;
        NewEntry^.edObject:=NewEntry^.edParent.edObject;
      end else

      if declaration.cxToken='repeat' then
      NewEntry^.edtype:=etRepeat else

      Begin
        (* method call? Make sure entry object supports this *)
        Result:=NewEntry^.edParent.edObject<>NIL;
        If Result then
        Begin
          (* check if object supports the method name *)
          Result:=RMLImplements(declaration.cxToken,NewEntry^.edParent.edObject);
          If Result then
          Begin
            (* Query object for method entry *)
            Result:=RMLGetMethodEntry
              (
              declaration.cxToken,
              NewEntry^.edParent.edObject,
              NewEntry^.edMethod
              );

            If result then
            Begin
              NewEntry^.edtype:=etMethod;
              NewEntry^.edObject:=NewEntry^.edParent.edObject;

              (* now parse the parameter conditions *)
              x:=0;
              While x<Length(declaration.cxCondition) do               Begin                 inc(x);                 If (declaration.cxCondition[x]=',')                 or (x=Length(declaration.cxCondition)) then                 Begin                   If x=Length(declaration.cxCondition) then                   FTemp:=FTemp + declaration.cxCondition[x];                   FTemp:=trim(FTemp);                   If length(FTemp)>0 then
                  Begin
                    (* create a new parameter *)
                    FLen:=length(NewEntry^.edParameters);
                    setlength(NewEntry^.edParameters,FLen+1);

                    New(FPar);

                    If RMLIsString(FTemp) then
                    FPar^.prType:=daString else
                    if RMLIsNumber(FTemp) then
                    FPar^.prType:=daNumber else
                    if RMLIsBoolean(FTemp) then
                    FPar^.prType:=daBoolean else
                    FPar^.prType:=daCustom;

                    Case FPar^.prType of
                    daString:
                      Begin
                        Delete(FTemp,1,1);
                        Delete(FTemp,length(FTemp),1);
                        FPar^.prSize:=Length(FTemp);
                        FPar^.prValue:=AllocMem(FPar^.prSize);
                        move(FTemp[1],FPar^.prValue^,FPar^.prSize);
                      end;
                    daNumber:
                      Begin
                        FPar^.prSize:=SizeOf(Integer);
                        FPar^.prValue:=AllocMem(FPar^.prSize);
                        PInteger(FPar^.prValue)^:=StrToInt(FTemp);
                      end;
                    daBoolean:
                      Begin
                      end;
                    daCustom:
                      Begin
                      end;
                    end;

                    NewEntry^.edParameters[FLen]:=FPar;
                    FTemp:='';
                  end else
                  Begin
                    //Invalid parameter error
                  end;
                end else
                FTemp:=FTemp + declaration.cxCondition[x]
              end;

              {
              Validate parameter datatypes here!
              If  (Length(NewEntry^.edParameters)>0) then
              Begin
                for x:=Low(NewEntry^.edParameters) to
                high(NewEntry^.edParameters) do
                Begin
                  newEntry^.edObject.
                end;
              end;  }

            end;

          end else
          Begin
            // property assignment test here
          end;
        end;
      end;

      (* Failed to tokenize? *)
      If not Result then
      Begin
        (* dispose of entry data *)
        Dispose(NewEntry);
        NewEntry:=NIL;
        SetLength(Root^.edSubEntries,FCount);
        Context.pcError:=Format('Invalid token "%s"',[declaration.cxToken]);
      end;

    end else
    Error:='AddEntry failed, root can not be NIL';
  end;

  Function  RMLParseObject(Var Context:TRMLParseContext;
            Const Root:PRMLEntryData):Boolean;
  var
    FChar:        Char;
    FTemp:        String;
    FDeclaration: TRMLEntryDeclaration;
    FNewEntry:    PRMLEntryData;
  Begin
    Result:=Context.pcSignature=SizeOf(Context);
    If result then
    Begin
      (* update cursor *)
      inc(Context.pcPos);
      inc(Context.pcCol);

      while Result and (Context.pcPos<Context.pcLen) do
      Begin
        FChar:=Context.pcData[Context.pcPos];
        Case FCHAR of
        #13:
          Begin
            inc(Context.pcRow);
            inc(Context.pcPos);
            Context.pcCol:=0;
            Continue;
          end;
        ';':
          Begin
            Result:=RMLParseEntry(trim(FTemp),FDeclaration,Context.pcError);
            If result then
            Result:=RMLAddEntry
              (
                Context,
                FDeclaration,
                root,
                FNewEntry,
                Context.pcError
              );

            If Result then
            Begin
              inc(Context.pcPos);
              inc(Context.pcCol);

              If FNewEntry^.edtype=etIF then
              Result:=RMLParseObject(Context,FNewEntry);
            end;
            FTemp:='';
          end;
        '{':
          Begin
            Result:=RMLParseEntry(FTemp,FDeclaration,Context.pcError);
            If Result then
            Begin
              Result:=FDeclaration.cxToken='if';
              If result then
              Begin
                Result:=RMLAddEntry
                  (
                  Context,
                  FDeclaration,
                  Root,
                  FNewEntry,
                  Context.pcError
                  );
                If Result then
                Result:=RMLParseObject(Context,FNewEntry);
              end;
            end;
            FTemp:='';
          end;
        '}':
          Begin
            inc(Context.pcCol);
            inc(Context.pcPos);
            Break;
          end;
        else
          Begin
            FTemp:=FTemp + FChar;
            inc(Context.pcCol);
            inc(Context.pcPos);
          end;
        end;
      end;
    end;
  end;

  Function RMLExecute(Const Context:TRMLParseContext):Boolean;

    Function RunEntry(Const Item:PRMLEntryData):Boolean;
    var
      FSubCount:  Integer;
      x:          Integer;
    Begin
      result:=True;

      Case Item^.edtype of
      etCreate,
      etAssignment:
        Begin

          FSubCount:=Length(Item^.edSubEntries);
          for x:=1 to FSubCount do
          Begin
            result:=RunEntry(Item^.edSubEntries[x-1]);
            If not result then
            break;
          end;

        end;
      etMethod:
        Begin
          {result:=RMLGetMethodEntry(Item^.edDeclaration.cxToken,
          Item^.edObject,FEntry);
          If result then
          result:=FEntry(Item); }
          Result:=TRMLProcEntry(Item^.edMethod)(Item);
        end;
      etRepeat:
        Begin
          //FSubCount:=Length(Item^.edSubEntries);
        end;
      etIf:
        Begin
          FSubCount:=Length(Item^.edSubEntries);
          for x:=1 to FSubCount do
          RunEntry(Item^.edSubEntries[x-1]);
        end;
      end;
    End;

  Begin
    Result:=Context.pcSignature=SizeOf(Context);
    If result then
    Begin
      result:=length(Context.pcError)<1;
      If result then
      result:=RunEntry(@Context.pcRoot);
    end;
  end;

  Function RMLCompile(Var Context:TRMLParseContext):Boolean;
  var
    FChar:        Char;
    FTemp:        String;
    FDeclaration: TRMLEntryDeclaration;
    FNewEntry:    PRMLEntryData;
  Begin
    Result:=Context.pcSignature=SizeOf(Context);
    If result then
    Begin
      Context.pcCol:=0;
      Context.pcRow:=0;
      Context.pcPos:=1;
      Context.pcError:='';

      while Result and (Context.pcPos<Context.pcLen) do
      Begin
        FChar:=Context.pcData[Context.pcPos];
        Case FCHAR of
        #13:
          Begin
            inc(Context.pcRow);
            inc(Context.pcPos);
            Context.pcCol:=0;
            Continue;
          end;
        '{':
          Begin
            Result:=RMLParseEntry(FTemp,FDeclaration,Context.pcError);
            If Result then
            Begin
              Result:=FDeclaration.cxToken='createobject';
              If result then
              Begin
                Result:=RMLAddEntry
                  (
                  Context,
                  FDeclaration,
                  @Context.pcRoot,
                  FNewEntry,
                  Context.pcError
                  );
                If Result then
                Result:=RMLParseObject(Context,FNewEntry);
              end;
            end;
            FTemp:='';
          end;
        '}':
          Begin
          end;
        else
          Begin
            FTemp:=FTemp + FChar;
            inc(Context.pcCol);
            inc(Context.pcPos);
          end;
        end;
      end;
    end;
  end;

  end.

Goodbye G+ Delphi Developers group

December 6, 2016 20 comments
A bit sad today. I typically post blog articles about Delphi, Smart Pascal and embedded projects both on Facebook and Google+, but apparently Smart Pascal is not welcome at Google’s Delphi group. Even examples where Delphi talks with a node.js server on a Linux box is apparently “off topic” and “not Delphi”. So much so that my post was deleted. I’m getting so sick and tired of this double standard. This is not the first time Smart is pushed aside. We were even banned from speaking at Delphi Tage a couple of years back. I really dont understand this attitude at all.
So easy, so powerful and you can deploy it anywhere. An embedded system, a dedicated server - or do a push to your Amazon / Azure cloud stack. Node.js is so powerful once you understand how to use it.

This is from the deleted post. If you look at the picture you will notice a Delphi UDP client/server on the right talking with a node.js service on the left. Aparently this is not Delphi enough for the admin in charge of the Delphi group on G+

I was particularly saddened by this since I have from time to time seen Elevate Software post about their web builder utility. But for some reason Smart pascal is not allowed to do the same (in this case it was a post from my blog). With all due respect for Elevate Software, but when it comes to interfacing Delphi with a modern IOT infrastructure or eco-system, Elevate is not even in the same ballpark. But focus here is on the technical, and Elevate’s web builder is ultimately in the same category as Smart Mobile Studio. So how their system that compiles a subset of object pascal to JavaScript is somehow allowed – while our product is not, I find both morally unacceptable and hypocritical. First of all since Embarcadero have held workshops in Angular.js (!).
Looking into the actual data, with JavaScript leading and JavaScript libraries on client and server side (Angular.js, Node.js) on the rise, it was nice to see that while Delphi was not listed as an option, it was the most typed entry in the “others” category -Source: Marco Cantu
Let me sum up a few facts:
  • Smart Mobile Studio is written 100% in Delphi
  • Smart Mobile Studio was created from scratch to compliment Delphi
  • Smart Mobile Studio supports Remobjects SDK out of the box
  • Smart Mobile Studio supports Embarcadero Datasnap out of the box
  • Smart Mobile Studio helps Delphi developers write the middleware or interfacing that sits between a native Delphi solutions and a customer’s node.js or purely web-based solution. This is 2016 after all.
  • Where a Delphi developer would previously have to decline a job offering because the customer’s existing infrastructure is based on node or emits data unsuitable for traditional Delphi components, developers can now relax and write the missing pieces in a Smart pascal, a dialect roughly 90% compatible with the language they know and love to begin with.
  • Smart Mobile Studio ships with a wast RTL that greatly simplify talking with Delphi. It also has a VCL inspired component hierarchy where more and more complex behavior is introduced vertically. This gives your codebase a depth which is very hard to achieve under vanilla JavaScript or Typescript.
  • Smart Mobile Studio is all about Delphi. It is even used to teach programming in the UK. Teenagers who by consequence and association is statistically more likely to buy Delphi as they mature.
Now I have no problem understanding or respecting the notion of single-topic groups. But having said that I expect the administrator(s) to apply this rule equally and without exception. Not just Smart pascal.
The reality of 2016 is that no Delphi developer use a single language or dialect. That may have been true 10-15 years ago, but that’s not where we are today. When a customer demands that your interface your Delphi software with their existing node.js or io based eco-system there are only 3 options available:
  • Decline the project
  • Learn JavaScript or Typescript and do battle with its absurd idiosyncrasies, lack of familiar data types, lack of inheritance and lack of everything you are used to
  • Use Smart Mobile Studio to write the middleware between your native solution and the customer existing infrastructure

If you pick option number two, it wont take many days before you realize just how alien JavaScript is compared to Delphi or C++ builder. And you will consequently start to understand the value of our RTL which is written to deal with anything from low-level coding (allocmem, reallocmem, fillmemory, move, buffers, direct memory access, streams and even threading). Our RTL is written to make the JavaScript virtual machine palatable to Delphi developers.

Banning dialects?

Once you start banning dialects of a language or an auxiliary utillity designed to empower Delphi and make sure developers can better interface with that aspect of the marketplace – where does it stop? I am curious to where exactly the Google+ Delphi group draws the line here to be honest.

Should Remobject Oxygene likewise be banned since it helps Delphi developers target the dot net framework? That would be odd since Delphi shipped Oxygene for years.

Should script engines be banned? What about SQL? SQL is a language most Delphi developers know and use – but it is by no measure object pascal and never will be. Interestingly, Angular.js seems to be just fine for the Google+ Delphi group. Which is a bit hypocritical since that is JavaScript plain and simple.

What about report engines? Take FastReport for instance: FastReport have for the past decade or more bolted their own scripting engine into the product, a script engine that supports a subset of object pascal but also visual basic (blasphemy!). On iOS you are, if we are going to follow the Apple license agreement down to the letter, not even allowed to use FastReport. Apple is very clear on this. Any applications that embed scripting engines and at the same time download runnable code (which would be the case when downloading report files) is not allowed on appstore. The idea here is that should some code be downloaded that is harmful, well then Apple will give you a world of hurt. And even if you consider it a report-file, it does contain runnable code – and that is a violation.

So is Fastreport Delphi enough? Or is that banned as well?

Where exactly do we draw the line? I can understand banning posts about dot net if it’s all about C#, but if it’s in relation to Delphi or deals with Delphi talking with dot net (or an actual dialect like Oxygene) then I really don’t see why it could be banned or deleted. Even Delphi itself uses dot net. It’s one of the pre-requisites for installing Delphi in the first place. I guess Delphi should also be banned from the Delphi group then?

In our group on Facebook, all are welcome. Embarcadero, Lazarus and Free Pascal, Elevate software (be it their database engines or web builder), Pax compiler, DWScript, Smart Pascal, NewPascal (or whatever it’s called these days) and even Turbo Pascal for that matter. And we sure as shit dont delete posts of Delphi talking to another system.

So goodbye G+ and good luck

Smart Pascal, supported server types

December 2, 2016 1 comment
Merry_XMAS

Use node.js to fill your xmas with fun!

Node.js is probably one of the coolest pieces of software I have had the pleasure to work with for years. It’s platform independent and available for just about every operative system you can imagine. I would even go so far as to say it has become universal.

NodeJS allows you not only to write server-side JavaScript, but also your own system level services. This is especially easy on Linux where the philosophy regarding a service is somewhat different from Microsoft Windows. On Linux, a simple bash script can be installed as a service. You can write services in python, perl or whatever tickles your fancy.

Our latest addition: UDP

Today I had a few minutes to test the UDP implementation I finished last week, and I was for some odd reason expecting an exception or “something” to come up. You know.. when something is that easy to write, there’s typically is a catch right? Or maybe im just and old, cynical Delphi developer. Either way, it worked on the first try!

So easy, so powerful and you can deploy it anywhere. An embedded system, a dedicated server - or do a push to your Amazon / Azure cloud stack. Node.js is so powerful once you understand how to use it.

Compiled SMS (node.js) talking with a Delphi application

Now I realize that UDP is not what you use for high-end, reliable communication. But that is beside the point. I want the code you get access to in our next update to be polished, easy to use and something you can rely on. And the keyword here is “co-operation”. My personal service stack that I host in my own home is written in 4 different languages. You have Delphi and C# services running under Windows, you have Lazarus daemons on my Linux box (a full PC) and last but not least — you have Smart Pascal servers running on embedded hardware.

Our list of server types now include:

  • HTTP
  • TCP
  • WebSocket
  • UDP

I use UDP more or less as a signal trigger between processes to handle wait, ready, update and even restart. So by broadcasting a single “shutdown” signal, all my machines will gracefully stop and then power down.

So, how does an UDP server look like? It borders on ridicules how little effort it takes:

procedure TNodeService1.SetupUDPServer;
var
  Server: TNJUDPServer;
begin

Server := TNJUDPServer.Create;
Server.Port := 1881;
Server.Address := '127.0.0.1';
Server.MulticastLoopback := true;
Server.Broadcast := true;
Server.Exclusive:= false;

Server.OnMessage := procedure (Sender: TObject; Data: variant;
          RequestInfo: TNJUDPRequestInfo)
begin
  writeln("Message recieved!");
end;

Server.OnError := procedure (Sender: TObject; ErrorObj: TJsErrorObject)
begin
  writeln("Error:" + ErrorObj.message);
end;

Server.OnClose := procedure (Sender: TObject; ErrorObj: TJsErrorObject)
begin
  writeln("Server closed");
end;

Server.OnAfterServerStarted := procedure (sender: TObject)
begin
  writelnF("Server started, listening on post %d @ %s",
  [Server.port, Server.Address]);
end;

Server.Active := true;
end;

That’s pretty much it. Naturally you have to fill in the blanks, but the above code is all you have to write to create a UDP server. The cool part is that all server classes inherit from a common ancestor, so once you know how to code one – you have a leg up on using the rest of them.

Running the server as a Linux Daemon

Like all languages, node.js has a few tools that are considered standard. It’s like we Delphi developers take component’s and libraries like GR32 and SynEdit for granted. These packages have become so standard that we forget how hard it can be for beginners to get an overview of what’s available.

Turns our node.js is no different, and the tool everyone is using to run their node.js code as a dedicated background service (meaning that it will start during the boot sequence) is called PM2 or node.js process manager v2.

Running your Smart Pascal server as a system-level daemon is very easy once you know what to look for :)

Running your Smart Pascal server as a system-level daemon is very easy once you know what to look for 🙂

In short, PM2 is like a watch-dog that keeps an eye on your service. If it crashed PM2 will re-started it (unless you tell it otherwise), it also does extensive logging for you – and it will even generate a startup script that you can add to the Linux (or windows) startup sequence.

But the most important aspect of PM2 is that you can easily get some “live” info on your services, like how much memory they consume, the CPU consumption and everything else. It’s also PM2 that makes it a snap to run your node.js servers in cluster mode, meaning that you can spread the workload over multiple machines and cores for better performance.

Getting PM2 up and running is easy enough. Just make sure you have installed NPM first, which is the “node package manager” front-end. To install NPM you just write:

sudo apt-get install npm -y

And with NPM available on your system, you install PM2 through NPM (PM2 is ofcourse written in node.js itself):

npm install pm2 -g

Next, having compiled our Smart Pascal project, we copy the file over to a shared folder on our raspberry PI (I installed Samba to make this easier), cd into the folder and type:

pm2 start backend/backend.js --name backend

Notice the “–name backend” part? PM2 allows you to assign names to your code. This makes it easier to manage them (not having to type the full path every single time).

To check if our service is now installed, type:

pm2 show backend

And voila! We have a live Smart Mobile Studio server running on a Raspberry PI 3! Now whip out Delphi and recycle those legacy services!

Delayed execution for Delphi

November 23, 2016 6 comments
The Dispatch class is made to mimic the behavior of TW3Dispatch in the Smart Pascal RTL, which is based on WC3 DOM behavior

The Dispatch class is made to mimic the behavior of TW3Dispatch in the Smart Pascal RTL, which is based on WC3 DOM behavior

One of the features I really miss when using Delphi, is TW3Dispatch from Smart Mobile Studio. It makes it so easy to schedule code to execute. In Delphi you either have to call the Windows API directly, use a TTimer or spawn a thread to achieve the same. Well now there is one for Delphi as well!

Now I did make a version of this library earlier which was VCL only (using the Windows API) but lately I decided to upgrade it to use threads – and thus the library is now platform independent and FMX compatible.

Delayed execution?

Ok let’s take a practical example. Let’s say you want to display a welcome form when your application starts; or actually – a few milliseconds after the main form has appeared on screen. How would you solve that?

Most people slap a TTimer on the main form, set a suitable delay and define Active as true. Then in the OnTimer() event they just disable the timer and do their thing.

Another practical example is ordinary button clicks. For most normal tasks the ordinary OnClick event works fine. But there are times where you may want to delay execution of “something” until after the OnClick event has finished. Especially if you are doing asynchronous calls to a server (for instance).

If you are going to use TTimer for stuff like that, even if you isolate it in functions, you are going to go insane sooner rather than later.

With my library it’s a one-liner:

TQTXDispatch.Execute( procedure ()
  begin
    StartDownload();
  end, 200);

Or, if you want to run through some steps you can use a repeater:

TQTXDispatch.RepeatExecute( function (): boolean
  begin
    result := DoNextStep;
  end, 200, 64); // repeat 64 times with 200ms interval

There is also a ExecuteEx() method that returns a handle, which you can then use to cancel a pending execution with AbortExecute().

Porting code between Smart and Delphi

While I can think of 100 uses alone for the library in Delphi, my primary motivation is making it easier to port code between Delphi and Smart. And TW3Dispatch can be tricky to “slap together” or “quickly adjust” to make code compile on both platforms. So hopefully this will help!

unit qtx.utils.callback;

//#############################################################################
//
// Unit:      Windows callback mechanism
// Author:    Jon Lennart Aasenden
// Copyright: Jon Lennart Aasenden, Quartex Components
// Website:   http://www.quartexcomponents.com
//#############################################################################
//
//  Purpose:
//    This unit provides methods for blocking delay of program execution,
//    and also a non-blocking callback function for anonymous methods.
//
//  The TQTXCallback.Delay() method is blocking, meaning that it will halt
//  program execution for the duration of the MS parameter.
//
//  The TQTXCallback.Callback() method is NON-blocking, meaning that it will
//  create a winAPI time object that, when the duration of MS is expired,
//  will execute the inline anonymous function.
//
//  NOTE: There are two overloaded versions: One for anonymous methods,
//        and another for object methods (when used within a class).
//
//  Example:
//     TQTXCallback.callback(
//        procedure (
//          begin
//            showmessage('2 seconds have gone by');
//          end,
//     2000);
//
//
//     TQTXCallback.delay(nil,4000);
//     showmessage('4 seconds have gone by');
//
//
//#############################################################################

interface

uses  System.SysUtils, System.Classes,
      Winapi.Windows, System.Generics.Collections;

// Toggle this to use WinAPI only, VCL restricted (!)
{$DEFINE USE_THREADS}

type
[weak]
TQTXObjectProc = procedure of object;
TQTXRepeatFunc = function (): boolean;

TQTXDispatch = Class
public
  (* Anonymous method implementation *)
  class procedure Execute(const aCallback: TProc; const ms: Cardinal); overload;
  class procedure Execute(const aCallback: TQTXObjectProc;
        const ms: Cardinal);Overload;

  class function  ExecuteEx(const aCallBack:TProc; const ms: Cardinal):THandle;
  class procedure AbortExecute(const aHandle: THandle);

  class procedure RepeatExecute(const Entrypoint: TQTXRepeatFunc; const ms: cardinal; Rounds: integer);

  class function GetTickCount: Cardinal;

  (* Object method implementation *)
  class procedure Delay(const aCallback: TProc; const ms: Cardinal);overload;
  class Procedure Delay(const aCallback: TQTXObjectProc;
        const ms:Cardinal);overload;
end;

implementation

{$IFNDEF FMX}
uses Vcl.Forms;
{$ELSE}
uses Fmx.Forms;
{$ENDIF}

type

TQTXDispatchThread = class(TThread)
strict private
  FCBAnon:  TThreadProcedure;
  FCBObj:   TQTXObjectProc;
  FDelay:   integer;
  FObjCall: boolean;
  fid: integer;
protected
  procedure ObjCallback;
  procedure Execute; override;
public
  property Id: integer read FId write FId;
  constructor Create(ms: integer; Callback: TThreadProcedure); overload;
  constructor Create(ms: integer; Callback: TQTXObjectProc); overload;
end;

TThreadDictionary = class(TDictionary<integer, TQTXDispatchThread>)
public
  procedure ThreadFinished(Sender: TObject);
end;

procedure TThreadDictionary.ThreadFinished(Sender: TObject);
var
  LId: integer;
begin
  if assigned(sender) then
  begin
    if assigned(self) then
    begin
      LId := TQTXDispatchThread(sender).Id;
      self.Remove(LId);
    end;
  end;
end;

var
{$IFDEF USE_THREADS}
_LUT: TThreadDictionary;
{$ELSE}
_LUT: TDictionary<UINT, TProc>;
{$ENDIF}
_TED: integer;

function QTXProgramEnded: boolean;
begin
  result := not assigned(application);
  if not result then
  result := application.terminated;
end;

//#############################################################################
// TQTXDispatchThread
//#############################################################################

constructor TQTXDispatchThread.Create(ms: integer; Callback: TQTXObjectProc);
begin
  inherited Create(true);
  FCBObj := Callback;
  FObjCall := true;
  FDelay := ms;
end;

constructor TQTXDispatchThread.Create(ms: integer; Callback: TThreadProcedure);
begin
  inherited Create(true);
  FCBAnon := Callback;
  FObjCall := false;
  FDelay := ms;
end;

procedure TQTXDispatchThread.ObjCallback;
begin
  if not QTXProgramEnded then
  FCBObj();
end;

procedure TQTXDispatchThread.Execute;
begin

  repeat
    sleep(1);
    dec(FDelay);
    if FDelay<1 then     begin       break;     end;   until terminated;   if not terminated then   begin     try       case FObjCall of       false:         begin           try             if assigned(FCBAnon) then             TThread.Queue(nil, FCBAnon);           except             on exception do;           end;         end;       true:         begin           try             Synchronize(ObjCallback);           except             on exception do;           end;         end;       end;     finally       terminate;     end;   end; end; //############################################################################# // TQTXDispatch //############################################################################# class function TQTXDispatch.GetTickCount: Cardinal; begin   result := TThread.GetTickCount; end; class procedure TQTXDispatch.RepeatExecute(const Entrypoint: TQTXRepeatFunc;   const ms: cardinal; Rounds: integer); begin   if assigned(Entrypoint) then   begin     if (ms >0) and (Rounds >0) then
    begin
      if not QTXProgramEnded then
      begin
        Execute(procedure ()
        begin
          if Entrypoint() then
          begin
            RepeatExecute(Entrypoint, ms, Rounds -1);
          end;
        end, ms);
      end;
    end;
  end;
end;

class procedure TQTXDispatch.Execute(const aCallback: TProc; const ms: Cardinal);
  {$IFNDEF USE_THREADS}
  procedure w3_invoke(hwnd: HWND; uMsg: UINT;
            idEvent: UINT_PTR;dwTime: DWORD);stdcall;
  var
    mProc:  TProc;
  begin
    KillTimer(0,idEvent);
    try
      if assigned(_LUT) then
      begin
        mproc:=_lut.Items[idEvent];
        _lut.Remove(idEvent);
        if assigned(mProc) then
        mproc;
      end;
    except
      on exception do;
    end;
  end;
  {$ENDIF}

{$IFDEF USE_THREADS}
var
  LThread: TQTXDispatchThread;
{$ENDIF}
begin
  if Assigned(_LUT) then
  {$IFDEF USE_THREADS}
  begin
    inc(_TED);
    LThread := TQTXDispatchThread.Create(ms,TThreadProcedure(aCallback));
    LThread.FreeOnTerminate := true;
    LThread.Id := _TED;
    LThread.OnTerminate := _LUT.ThreadFinished;
    _LUT.Add(_TED, LThread);
    LThread.Start;
  end;
  {$ELSE}
  _LUT.add(SetTimer(0,0,ms,@w3_invoke),aCallback);
  {$ENDIF}
end;

class function TQTXDispatch.ExecuteEx(const aCallBack: TProc;
  const ms: Cardinal): THandle;

 {$IFNDEF USE_THREADS}
  procedure w3_invoke(hwnd: HWND; uMsg: UINT;
            idEvent: UINT_PTR;dwTime: DWORD);stdcall;
  var
    mProc:  TProc;
  begin
    KillTimer(0,idEvent);
    try
      if assigned(_LUT) then
      begin
        mproc:=_lut.Items[idEvent];
        _lut.Remove(idEvent);
        if assigned(mProc) then
        mproc;
      end;
    except
      on exception do;
    end;
  end;
  {$ENDIF}

{$IFDEF USE_THREADS}
var
  LThread: TQTXDispatchThread;
{$ENDIF}
begin
  result := 0;
  if Assigned(_LUT) then
  begin
    {$IFDEF USE_THREADS}
    inc(_TED);
    result := _TED;

    LThread := TQTXDispatchThread.Create(ms, TThreadProcedure(aCallback));
    LThread.FreeOnTerminate := true;
    LThread.Id := result;
    LThread.OnTerminate := _LUT.ThreadFinished;
    _LUT.Add(result, LThread);
    LThread.Start;
    {$ELSE}
    result := SetTimer(0,0,ms,@w3_invoke);
    _LUT.Add(result,aCallback);
    {$ENDIF}
  end;
end;

class procedure TQTXDispatch.AbortExecute(const aHandle: THandle);
var
  LInstance: TQTXDispatchThread;
begin
  if assigned(_LUT)
  and _LUT.ContainsKey(aHandle) then
  begin
  {$IFDEF USE_THREADS}
    if _lut.TryGetValue(aHandle, LInstance) then
    begin
      if not LInstance.Terminated then
      LInstance.Terminate;
    end;
  {$ELSE}
    _lut.Remove(aHandle);
    KillTimer(0,aHandle);
  {$ENDIF}
  end;
end;

class procedure TQTXDispatch.Execute(const aCallback: TQTXObjectProc;
  const ms: cardinal);

  {$IFNDEF USE_THREADS}
  procedure w3_invoke(hwnd: HWND; uMsg: UINT;
            idEvent: UINT_PTR;dwTime: DWORD);stdcall;
  var
    mProc:  TProc;
  begin
    KillTimer(0,idEvent);
    try
      if assigned(_LUT) then
      begin
        mproc:=_lut.Items[idEvent];
        _lut.Remove(idEvent);
        if assigned(mProc) then
        mproc;
      end;
    except
      on exception do;
    end;
  end;
  {$ENDIF}

{$IFDEF USE_THREADS}
var
LThread: TQTXDispatchThread;
{$endif}
begin
  if Assigned(_LUT) then
  {$IFDEF USE_THREADS}
  begin
    LThread := TQTXDispatchThread.Create(ms, aCallback);
    LThread.FreeOnTerminate := true;
    LThread.OnTerminate := _LUT.ThreadFinished;
    inc(_TED);
    LThread.Id := _TED;
    _LUT.Add(LThread.Id, LThread);
    LThread.Start;
  end;
  {$ELSE}
  _LUT.add(SetTimer(0,0,ms,@w3_invoke),aCallback);
  {$ENDIF}
end;

class procedure TQTXDispatch.Delay(const aCallback: TProc;
      const ms: cardinal);
var
  LThen:  DWord;
begin
  if (not QTXProgramEnded) and (ms > 0) then
  begin
    try
      LThen := self.GetTickCount + ms;

      repeat
        Sleep(1);
        if QTXProgramEnded then
        break;
      until ( self.GetTickCount >= LThen );

      if assigned(aCallback) and (not QTXProgramEnded) then
      begin
        try
          aCallback;
        except
          on exception do;
        end;
      end;
    except
      on exception do;
    end;
  end;
end;

class procedure TQTXDispatch.Delay(const aCallback: TQTXObjectProc;
      const ms:Cardinal);
var
  LThen:  DWord;
begin
  if (not QTXProgramEnded) and (ms > 0) then
  begin
    try
      LThen := self.GetTickCount + ms;
      repeat
        Sleep(1);
        if QTXProgramEnded then
        break;
      until ( self.GetTickCount >= LThen );

      if assigned(aCallback) and (not QTXProgramEnded) then
      begin
        try
          aCallback;
        except
          on exception do;
        end;
      end;
    except
      on exception do;
    end;
  end;
end;

initialization
begin
{$IFDEF USE_THREADS}
  _LUT := TThreadDictionary.Create;
{$ELSE}
  _LUT := TDictionary<UINT,TProc>.Create;
{$ENDIF}
end;

finalization
begin
  if Assigned(_LUT) then
  FreeAndNil(_LUT);
end;

end.

Overclocking the Raspberry PI 3

November 17, 2016 2 comments

On the menu for today was a meeting with my partner in crime, Glenn, who is working hard on the Raspberry PI Linux distro for Smart Mobile Studio – and then do some Linux stuff. For the Smart Pascal headers to actually fit the latest stuff I needed (doh) the latest version of Node to test on. Which has to be built from C/C++ source code on the Arm device.

Building purely from C/C++ source on a PI is .. probably the worst thing I did all day. Cross-compile from your PC instead

Building purely from C/C++ source on a PI is .. probably the worst thing I did all day. Cross-compile from your PC instead

So I baked a fresh copy of Raspbian with the sexy new Pixel UI, updated packages and so on. I set the CPU in performance mode (yes the CPU has different modes) before doing a compile — installed git, cloned out the latest nodejs repository and hit configure + make.

Well, its been 60+ minutes since the bloody build started so I thought, hey why not overclock the sucker and at least shave some off the waiting. But sadly overclocking is not supported by the official Raspbian system settings.

After a bit of google time I found a guy that had gone through the ropes and settled on a sexy 1400 frequency overclock with an over-voltage at around 6. This is stable and the core temp when spawning 4 intense 100% threads (one for each core) is around 58-60 degrees. In other words, well within the specifications (the cpu blows at around 80 degrees and will also kick the bucket at -40).

15102074_10153963465515906_1574367388_o

The dark side of the force is a pathway to many abilities, some considered unnatural

So — you do this at your own risk. Do not attempt this unless you understand the risk. By overclocking you can kiss the warranty good-bye, and I take NO RESPONSEBILITY for whatever damage or loss this could cause. You are on your own when you overclock anything, be it your dishwasher, PC or Raspberry PI embedded boards.

Having said that, the worst that can happen is that you kill your PI. Not exactly a heart breaking loss since they retail at around $35. In most cases when a CPU gets to warm it just shuts down or programs crash. So should that happen let it cool, then use a normal image and leave overclocking alone.

Heat sinks

Yes it may look silly, but if you overclock anything you need to delegate the extra heat somewhere. You can get a heat-sink set for around $1-$3 depending on the vendor. I ordered a couple of sets from Asia at around $2. You get the same kit at RC components for roughly the same price.

Tiny little things with double-sided sticky tape. Works like a charm.

Tiny little things with double-sided sticky tape. Works like a charm.

Ironically it’s not just the CPU that overheat, but also the USB controller and WIFI chip. Thats why a trim-set typically ships with 3 mini heat sinks.

Let there be light!

Boot your PI with a normal raspberry image (and take a backup in case of karma). When you get to the desktop, open a command-prompt and cd all the way back to root (/). Then cd into the boot folder. Inside the boot folder you will find a file called “config.txt”. You need to edit this file with admin rights:

sudo nano config.txt

This opens up nano, a dos like text editor. Just scroll down until you find:

#uncomment to overclock the arm. 700 MHz is the default.
#arm_freq=800

Change that to the following:

#uncomment to overclock the arm. 700 MHz is the default.
arm_freq=1400
over_voltage=6

hit CTRL+x, when asked if you want to save – hit “y”. The file is now saved and the next time you boot – it should run at 1.45 GHz. Which is quite nice

To reboot, just type

reboot

This is the setting that works best for me:

#uncomment to overclock the arm. 700 MHz is the default.
arm_freq=1350
over_voltage=5
sdram_freq=500
gpu_freq=500

The sdram overclocking might not work at all (did for me though), and some PI’s actually wont run any faster above arm_freq=1300, but mine works fine with the above settings.

From 3 FPS to 7 FPS! We are talking millions of pixels per frame, and going from 3 to 7 FPS is quite a boost!

Click for video!!! From 3 FPS to 7 FPS! We are talking millions of pixels per frame, and going from 3 to 7 FPS is quite a boost!

Also note that if it wont boot, just plug the sd-card into your PC and edit the config.txt there in notepad (the boot folder on the pi is actually a mapping to the windows fat32 partition).

Remember to fit the heat-sinks BEFORE you boot (!)

Updated

Since some guy has posted “you dont understand the raspberry PI” in the comment section I just want to respond to that briefly.

Yes, you are partly right, there are aspects of the PI I dont have a clue about. Nor do I pretend to be guru about every piece of tech i come into contact with (that would be a very booring life IMHO). In fact, what I think is exciting about that little machine is that you can tinker and learn as you go – and that even mid-level users like myself can get results.

The JavaScript demo I used to test the clocking is actually a reasonable starting point. At first it was running at roughly 1 fps (due to poor coding, I used timers rather than RequestAnimationFrame to synchronize redraw). By just changing the arm_freq to 1350 this doubled, to a whopping 2 fps.

After i tweaked the memory clock-speed and gpu and set some overvoltage, that in turn went up to a stable 7 fps.

Since this is a per-pixel demo, meaning that it uses the canvas to draw stuff, with a secondary callback cycle applying an alpha clear to give the impression of fading pixels out ( fillrect(0,0,width,height, rgba(0,0,0,0.3)) ) running out of sync — the demo is actually processing millions of bytes per frame.

Just for sake of argument, lets say the chrome window is “1024 x 1024” in 32 bit mode (which is probably a bit small, but lets go with that). We then get:

  • 1024 * 4 (four bytes per 32 bit pixel) = 4096 bytes per scanline
  • 4096 * 1024 = 4,194304 million bytes per frame

There is also a stride offset, but I will ignore that since it probably amounts to about 1kb or less. Right, since we are now drawing 7 frames instead of 3, this gives us:

4 * 4194304 = 16,777216 – a boost of 16.7 million bytes per second

20161117_214401

I may not be a Raspberry PI guru, nor would I pretend to be one, but boosting the javascript virtual machine by that magnitude is pretty good. And remember: JavaScript is single threaded. I have yet to see what happens when we use all the cores here.

But I am open for information and would love to hear your take on this. Im sure we can dig down and find out exactly which of my settings had the most impact.

Either way: overclocking and tuning the Raspberry PI 3 for better performance is possible. And that was the only thing on the table here.

Sneak peek at the new Smart RTL – Part 2

November 7, 2016 1 comment

If you missed the last sneak peek, click here to catch the first part.

In the previous post we had a look at a handful of the new features of the upcoming RTL. This time we will look at a couple of more. There is quite a few to write about, so I could probably do 10 of these “peek posts” to be honest.

Fine tune control

Support for CreationFlags is something we finally got into the RTL. If you have created your own controls using Delphi or Lazarus you are probably familiar with them? In short, it’s a set of special values that defines how a control should behave. And they are very important.

CreationFlags is defined as a class function, so this is a method you override when defining your own controls. It looks like this:

class function TW3TagObj.CreationFlags:TW3CreationFlags;
begin
  result := [
    cfAllowSelection,
    cfSupportAdjustment,
    cfReportChildAddition,
    cfReportChildRemoval,
    cfReportMovement,
    cfReportReSize
    ];
end;

The flags themselves are defined as such:

  TW3CreationFlags = set of
    (
    cfIgnoreReadyState,     // Ignore waiting for readystate
    cfSupportAdjustment,    // Controls requires boxing adjustment (default!)
    cfReportChildAddition,  // Dont call ChildAdded() on element insertion
    cfReportChildRemoval,   // Dont call ChildRemoved() on element removal
    cfReportMovement,       // Report movements? Managed call to Moved()
    cfReportResize,         // Report resize? Manages call to Resize()
    cfAllowSelection,       // Allow for text selection
    cfKeyCapture            // assign tabindex, causing the control to issue key and focus events
    );

So what is the big deal? Well, lets say you want keyboard support for a control (and under Smart Pascal you are expected to write your own controls all the time, it’s a part of how you work with the RTL. Just like C# expects you to inherit and implement your own classes) — now previously you could only capture input from traditional input controls. Like a Textbox or Memo. Which is quite limiting to say the least.

Now however, all you have to do to get keyboard support is to add cfKeyCapture and the OnKeyPress (and associated events) will fire, no matter if the control is based on a DIV (standard) or some other HTML element.

win8cat

The Windows 10 category header is fully animated. It can also be navigated using the arrow keys.

As you can see from the list of flags you can now disable some of the inner mechanics for a control if you don’t need it. Things like automatic calls to Resize() whenever the width or height is changed, the call to Moved() whenever a control is (drumroll) moved can also be disabled.

If you are wondering why this should be good news then consider this: Imagine you are implementing a really cool DB grid in Smart Pascal. One that needs to display 10.000 records just as fast as 100 records. When dealing with presentation of this amount of items, every call your code makes count (!) In fact, fast and responsive grids has been something of a holy grail for JavaScript over the years. Its only in the past 3 years we are starting to see high quality grids that is en-par with their native counterparts (like Developer Express, TMS and others).

Just like in Delphi or C# there are functions that the average developer don’t really care about or have to know to use the product – so its is with Smart. There will be functions that makes little sense to the average developer, but who are super important to people who make and sell components.

Ready, set, Action!

The core unit for visual controls in the RTL, SmartCL.Components.pas, have a number of additions, but the most important one is without a doubt the support for Actions. A feature of Delphi that has distinguished it from other RAD tools for decades (C# only recently got support for this, believe it or not).

But Actions is not so simple as you may think. It’s not just a matter of adding an Action property to our control-base. Actions also have to be reflected in the IDE, and it brings with it another feature that has been on our wish-list for a while now: namely non-visible controls!

  TW3TagObj = class(TW3CustomWidget, IW3ActionSubscriber)
  private
    FOwner:     TControlHandle;
    FHandle:    TControlHandle;
    FTagId:     string;
    FUpdating:  integer;
    FAttributes: TW3ElementAttributes;
    FComponentState:  TComponentState;

    {$IFDEF USE_ACTIONS}
    FAction:  TW3Action;
    function GetActionObject: TW3Action;
    procedure SetActionObject(const ActionObj: TW3Action);

    (* Implements:: IW3ActionSubscriber interface *)
    procedure RegisterAction(const Action: TW3Action);
    procedure UnRegisterAction(const Action: TW3Action);
    procedure ActionStateChanged(const Action: TW3Action);
    {$ENDIF}

Notice how TW3TagObj suddenly have a new ancestor? It used to inherit directly from TObject, but now suddenly it inherits from TW3CustomWidget (name is not yet carved in stone, we may call it something else since “widget” is often used for visual elements under QT C++). So indeed — non visual, drag & drop components is finally in the making.

But rather than cluttering up your forms we will go for a visual-studio like drop-zone at the bottom of your forms. This does a world of good for the size of your binaries. First of all, had we moved non-visual controls up to TComponent like Delphi and Lazarus do, it would mean a lot of useless code would be involved. TW3Component is designed to deal with child elements, child elements that have handles or map to the document object model. To avoid bloat we have introduced non-visual code earlier in the inheritance chain. Most non visual controls dont create or map to handles, nor do they create visual elements at all, so why have all that extra code attached to it?

GPU power

As you probably know, modern browser engines like Webkit are GPU powered. This means that a DIV or PRE tag (which our controls create when you call the constructor) can be marked for hardware acceleration. This was earlier a manual thing, something you would add to the stylesheet of your application.

Well, you can now call the method TW3TagObj.SetInitialTransformationStyles() in your constructor, and the control is marked for GPU rendering. Just remember: to much of a good thing is not cool. So think about what you accelerate. You would for example, not call this function on a DB grid, but rather on the rows being displayed.

Controllers

This is a new concept for Smart. A controller is a class that allows you to add extra features to a component indirectly (by proxy. Even existing controls). A controller is typically small and aims at delivering specialized features. Stuff that would be overkill to place in a TW3Widget (non visual control), yet to large for a utility unit. Since its all about control over something, we just let it name itself.

The base-class for a controller is very lightweight:

  TW3Controller = class(TW3OwnedObject, IW3Controller, IW3ControllerEventAccess)
  private
    (* IW3Controller Implementation *)
    function GetAttachedEventHandler:TW3ControllerAttachedEvent;
    procedure SetAttachedEventHandler(const EventHandler: TW3ControllerAttachedEvent);
    function GetDetachedEventHandler:TW3ControllerDetachedEvent;
    procedure SetDetachedEventHandler(const EventHandler: TW3ControllerDetachedEvent);
  protected
    function  QueryAttachment:Boolean;virtual;abstract;
    procedure DoAttach;virtual;
    procedure DoDetach;virtual;

    property OnAttached:TW3ControllerAttachedEvent;
    property OnDetached:TW3ControllerDetachedEvent;
  public
    Property  AutoDetach:Boolean;
    property  Attached:Boolean read QueryAttachment;
    procedure Attach(const AOwner: TObject);virtual;
    procedure Detach;virtual;

    constructor Create(AOwner: TObject);override;
    destructor Destroy;Override;
  end;

As of writing we have these controllers implemented:

  • Edge Sense
  • Swipe
  • Attributes
  • PlainScroll
  • MomentumScroll
  • iScroll

Edge-sense adds code to sensing when the mousepointer is within range of a controls edge. This controller is what you would use to deal with live resizing. It’s actually used in our new DB grid to deal with resize of column headers. As a controller it can be created on the fly and just attached to any custom control. It targets the handle and works with it from the JavaScript side of things, so it doesn’t interfere or override any code in the control it targets.

Our swipe-controller is a simple way to deal with horizontal or vertical swipes. The most common gestures on mobile devices. So if you are displaying information that can be swiped – like pages in a book, or scrolling in a new item — then this is a quick and easy solution. The alternative is to deal with gestures through events. But if all you need is swipe left/right or top/bottom (or both) – this controller is the ticket.

The scroll controllers are very cool. Let’s say you have a normal listbox right. Well, now you can chose what type of scrolling that listbox should use. If you want vanilla scrolling without any momentum, then you just attach a TW3PlainScrollController and it will scroll without any bouncing or acceleration. If you want acceleration, you create a TW3MomentumScrollController and attach that. Then it will scroll like an “iPhone list” (more or less). We have also wrapped iScroll, which is a JS library that emulates iPhone scrolling down to the letter (and then some).

Attributes allows you to store information in a tag. When you create a visual control, the RTL creates a tag via JavaScript. Traditionally these tags can only contain attributes they support. A while back this was extended so you can now add as many attributes as you like to a tag, providing it starts with “db-“. What is the use for this you ponder? Well let’s say you need some way of marking a control as active, or being processed. Rather than having to keep track of that state in a dictionary or array — you can now just mark the html tag directly.

We use this in our SmartCL.effects.pas unit. When you apply an effect to a control, the control is marked as “busy” while the effect is running. That way you don’t accidentally execute more than one effect on the same control (which would have unpredictable results). Instead the framework will cache up the effects you have applied, and execute them in sequence. And all of it is very efficient since the info is stored in the tag itself rather than in a JavaScript array of dictionary.

Mousewheel support

Indeed another tiny but very important feature! This has been added to TW3Scrollbar and thus all our scrolling controls now support this out of the box.

Any new components?

Oh yes, there is plenty of new components and quite a few JavaScript libraries for you to enjoy. Pretty much all of the QTX components have been given an overhaul and is now a part of the standard library. This includes:

  • TW3SimpleLabel
  • TW3EdgeNavigation
  • TW3CategoryHeader
  • TW3DBGrid
  • TW3DesignSurface
  • .. and much more (!)
Need a design surface? Perhaps you need to display a document, or create a real form designer? Well inherit from this and style it to your wishes

Need a design surface? Perhaps you need to display a document, or create a real form designer? Well inherit from this and style it to your wishes

Some of the more awesome libraries we have ported over are:

Wait a minute, ACE? What is that?

Ace is the #1 Javascript code editor, implemented in JavaScript (naturally). Basically it does the exact same thing that SynEdit or TBoneCode editor does for Delphi. Except that it does more than SynEdit, have better support for different languages and dialects, syntax highlighting, code proposal, code folding — all of it! And now it’s a control you can drop on a form.

Just to demonstrate what you can do with Smart Mobile Studio, we created a real compiler. A compiler that takes the legendary Amos Basic language of the 90’s, and generates assembly bytecodes for a virtual machine. All of it written in Smart Mobile Studio. Fancy a bit of retro game coding? Well at least the editor wont hold you back!

Amos Basic for the Amiga, recreated in HTML5. And it generates bytecodes as well! Programs can be executed both in the browser and for node.js

Amos Basic for the Amiga, recreated in HTML5. And it generates bytecodes as well! Programs can be executed both in the browser and for node.js

Pixie? Eh.. fairy dust much?

You can say that. Actually its the #1 multimedia content engine for JavaScript. It uses WebGL to render 2D graphics at incredible speed and with features normally found only in high-end game engines. Want to impress someone with your Smart skills? Whip out Pixie and knock ’em dead with effects they wont even believe is coming out of JavaScript!

You really should see this live, so pay them a visit and get blown away by the power you can now control from pascal!

You really should see this live, so pay them a visit and get blown away by the power you can now control from pascal!

Is there more?

Hell yeah, I have just scratched the surface! But my fingers hurt now so I’ll take a break and post more inside information in the days and weeks to come.

Stay tuned!

require.js for Smart Pascal

October 26, 2016 5 comments

I guess it had to happen. And why not? If you don’t know what require.js is then you can read up on it here: http://requirejs.org/ and also check out their repo on github. I’ll show you how to get this going for Smart Mobile Studio.

Require.js

Ever looked at a node.js program? Notice how they always start with require? It’s just all over the place. Well fret ye not, it’s basically node’s equivalent of pascal’s uses. Except, it also deals with code modularization, resolving dependencies and file-loading at the same time.

I don’t know what’s up with these JS developers. Brilliant coders and I love them for it, but they have this tendency to just stuff everything under the sun including the kitchen sink into a single package. This is what happens when they replace C and Pascal at universities around the world 🙂

require.js can be divided into 4 aspects:

  • Defining dependencies that must be in place before a piece of code executes
  • Loading JavaScript files en-mass and getting a nice callback when the files are ready
  • Loading of code modules, ala DLL files in a clever way
  • Support for plugins, there is a whole bunch available (check the website)

Loading JavaScript files en-mass

To be honest the VJL already have this. It’s been a part of our RTL for ages now. If you look at the unit SmartCL.FileUtils.pas you will find a class called TW3Storage sporting routines for loading not just scripts -but images, text-files, sound files, XML and even CSS. So in your face require.js! Ha!

It has the following interface:

  TW3Storage = static class
  public
    class procedure LoadXML(aFilename:String;
          const OnComplete:TW3XMLDataReadyEvent);

    class procedure LoadFile(aFilename:String;
          const OnComplete:TW3TextDataReadyEvent);overload;

    class procedure LoadFile(aFilename:String;
          const OnComplete:TW3StreamDataReadyEvent);overload;

    class function LoadCSS(const aRel,aHref:String;
         const OnComplete:TProcedureRef):THandle;overload;
    class function LoadCSS(const aRel,aHref:String):THandle;overload;

    class Procedure LoadScript(aFilename:String;
          const OnComplete:TProcedureRef);overload;
    class procedure LoadScript(aFilename:String);overload;

    class function LoadImage(aFilename:String;
          const OnComplete:TProcedureRef):THandle;overload;
    class function LoadImage(aFilename:String):THandle;overload;

    (* This function is for batch-loading an array of images.
       The callback will be invoked when all images are successfully loaded *)
    class Procedure BatchLoadImages(aFileNames:TStrArray;
          const OnComplete:TProcedureRef);
  End;

So we have little need for require.js when it comes to loading files. There is actually very little require.js (on the surface) has to offer the Smart RTL. Having said that – require.js is not just about loading files. It also deals with module declarations and dependencies when loading these – which is what a Delphi programmer would call a DLL file.

So I think it’s wise to include it for the future, because pure JS libraries is something we will be adding to the codegen later. I wont set a date for it, but I have already added quite a bit to the codegen.

As a bonus, the keyword “require” is already marked as a reserved keyword for node.js, so the syntax highlighter in Smart Mobile Studio (im testing this on version 2.2.1.4542 which is older than the current release) will highlight the keyword as an intrinsic procedure in SmartCL based projects as well. Neat and fits like a glove!

requirejs

Require is a reserved word in node.js

Right. Loading files is well and dandy, but what about the modules? I havent really bothered with that just yet. It’s there, and it shouldnt be to hard for a crafty Smart coder to complete it, but I want it to remain dormant until we have time to make it an integral part of Smart. No point in adding something super-cool and just expose it in the IDE. And we have our work cut out for us as it is (!) So let’s finish the backlog before we start with the DLL standard.

Also, Smart is package based just like Delphi and Lazarus, so the whole JavaScript “module” system is a bit of a mess really. It works, but elegant is not a word I’ll use to describe some of the JavaScript solutions out there.

You want it now? Ok, here you go!

Yes, this time you can actually get it now. I have written the wrapper in such a way that you can just download require.js, stuff it in a library, then add a unit to your RTL folder. I know I have irritated you with all these previews. Since this unit have no special dependencies you can assemble it in 10 minutes or less.

Click here to download both the test project and the library files if you don’t want to follow the tutorial.

First, download require.js and put that in the  $root\libraries folder. like this ([D] = Directory, [F] = file):

    • [D] Libraries
      • [D] Require.js
        • [F] require.js
requirejs_minified

You want the minified version so click that

For simplicity just rename the JavaScript file “require.js”, then create a text file in the same folder with the download url and the version number. That way you can keep track of the version when you update in the future.

Now let’s write some pascal to make this puppy conform to our way of doing things! But always remember, this file works with the document object model (DOM) and must never be used in a node.js application. Node has its own version of require(). So this belongs in SmartCL namespace and visual applications only (!).

Note: Normally I would urge you to save this file to the RTL folder, but that is quite risky. The RTL folder is under the control of our automatic update program (see license) and as such it can be deleted or replaced without further notice.
To avoid any changes you make being erased, place it side-by-side with the pascal file in $Libraries\require.js\SmartCL.Require.pas. Just saying.

Note: This file is copyright, but if you own Smart Mobile Studio you have the right to use it in your projects. If not, visit www.smartmobilestudio.com for licensing options.

{ **************************************************************************** }
{                                                                              }
{ Smart Mobile Studio - Runtime Library                                        }
{                                                                              }
{ Copyright (c) The Smart Company AS. All rights reserved.                     }
{                                                                              }
{ **************************************************************************** }
unit SmartCL.Require;

interface

uses W3C.DOM, System.Types, system.reader;

type

EW3RequireJS = class(EW3Exception);

TW3RequireError = class external (JDomError)
public
  property columnNumber: integer;
  property lineNumber: integer;
  property fileName: string;
  property message: string;
  property name: string;
  property stack: TStrArray;
  property requireType: string;
  property requireModules: TStrArray;
end;

TW3RequireErrHandler = procedure (err: TW3RequireError);

TW3RequireJSConfig = class external "requirejs.config"
  property enforceDefine: boolean;
  property baseUrl: string;
  property paths[name: string]: variant;
  property waitSeconds: integer;
end;

TW3RequireJS = class external "requirejs"
  property config: TW3RequireJSConfig;
  property onError: TW3RequireErrHandler;
end;

function Require: TW3RequireJS; overload;
procedure Require(Files: TStrArray); overload;
procedure Require(Files: TStrArray; const Success: TProcedureRef); overload;
procedure Require(Files: TStrArray; const Success: TProcedureRef;
  const Failure: TW3RequireErrHandler); overload;

implementation

{$R "require.js"}

function Require: TW3RequireJS;
begin
  try
    asm
    @result = require;
    end;
  except
    on e: exception do
    raise EW3RequireJS.Create({$I %FUNCTION%}, nil, e.message);
  end;
end;

procedure Require(Files: Array of string);
begin
  try
    asm
    require(@files);
    end;
  except
    on e: exception do
    raise EW3RequireJS.Create({$I %FUNCTION%}, nil, e.message);
  end;
end;

procedure Require(Files: TStrArray; const Success: TProcedureRef);
begin
  try
    asm
      require(@Files, @Success);
    end;
  except
    on e: exception do
    raise EW3RequireJS.Create({$I %FUNCTION%}, nil, e.message);
  end;
end;

procedure Require(Files: TStrArray; const Success: TProcedureRef;
  const Failure: TW3RequireErrHandler);
begin
  try
    asm
    require(@Files, @Success, @Failure);
    end;
  except
    on e: exception do
    raise EW3RequireJS.Create({$I %FUNCTION%}, nil, e.message);
  end;
end;

initialization
begin
  // When you compile a smart program, it will copy all files imported with
  // the $R compiler define, these will be stored in the $AppName/Res folder
  // and are loaded automatically.
  // Since that is the place you want to store other scripts as well, we
  // create an alias for the location. So $scripts will always point to
  // that path. You can add as many aliases you wish
  require.config.enforceDefine := false;
  require.config.paths['$scripts'] := '/res/';
end;

end.

Now save this as described above and give it a testdrive!
Start a new visual project, save the project before you start coding (important!)
Add a button, save once more (I know, I know.. this is being fixed now. I hate that bug so much). Double click the button in the designer, then add this:

procedure TForm1.W3Button1Click(Sender: TObject);
begin
  require(["https://ajax.googleapis.com/ajax/libs/jquery/3.1.0/jquery.min.js"],
      procedure ()
      begin
        showmessage("JQuery loaded");
      end);
end;

The above code will load jQuery (latest build) from google’s public API server. To make sure that everything went ok, click the “devtools” button in the Smart display browser and investigate:

Require() works like a charm :)

Require.js works like a charm 🙂

Note: JQuery is here just used as an example. You have absolutely no need for it under Smart, because the way visual controls work in the VJL a control always knows it’s childrens handles. So searching (hence the “query” in jquery) for tags and groups of elements have little meaning for object pascal in the browser. It already knows its children and have direct access to them. So no querying required.

Well — enjoy!

Why C# coders should shut up about Delphi

October 18, 2016 Comments off

EDIT: Some 3 years after this satire post was published, a website called “.NetRocks” decided to make a number out of it. The satire clearly went over their heads, and the facts I outline in the post was met with ridicule. I’m a bit disappointed with .NetRocks, because this could have been a great opportunity for them to learn more about modern object-pascal development and it’s ecosystems. They seem genuinely baffled that Delphi and Object-Pascal in general was being used in 2016 (and 2019), and their behavior and response demonstrates the very arrogance and ignorance we have come to expect from “brogrammers” of their type.

I must admit that I’m somewhat shocked that a massive show like .NetRocks is utterly ignorant of the fact that Object-Pascal as a language has millions of users around the world. You have compilers like Oxygene from RemObjects that targets .Net, Java, x86, ARM and WebAssembly. So their whole podcast on “how to port Delphi applications to C#” is absurd, because that’s not how you migrate code from Delphi to .Net. The Elements ecosystem, which covers C#, Pascal, Java, Swift and GoLang installs straight into Visual Studio and adds Object-Pascal to the environment. You can also convert source-code between Pascal and C# through automation. Manually going about such tasks is amateurish to say the least, but I honestly didn’t expect much when they open their podcast with verbal abuse.

vspas

While .NetRocks is manually porting pascal to C#, the rest of us use the Elements compiler and just compile the code to .Net Assemblies. You can also convert the code to any of the 5 languages Elements support. I have to wonder what rock .NetRocks lives under to be oblivious to this

Another point .NetRocks seem to have missed, is that Delphi has been aggressively optimized since Embarcadero took over from Borland. In 2016 Object Pascal ranked as #14 [2.164 score] on the Tiobi index over most popular programming languages [globally], just nine points below C#. So the most cost effective way of getting your legacy Delphi application modernized, is to just buy Elements from RemObjects, or upgrade your Delphi version to this century.

If they had bothered to check they would also have learned that I manage some 16+ developer groups on social media, with roughly 50.000 developers in total. I have also worked for Embarcadero that makes Delphi, so my statements while whimsical and satirical, were based on facts. I work with Fortune 500 companies in the US, as well as large companies around Europe, that all rely on Object-Pascal for their daily operations. Heck, even the Walt Disney company wanted to get rid of .Net in favour of Object Pascal in 2018. Turns out JIT is a disaster when coding against hardware interrupts and “old school” automation. Who could have guessed? (that was satire btw).

vsdelphi

Some people seem to think that Delphi ended with version 7 back in  the 90s. Delphi has been under heavy development since Embarcadero took over for Borland. .NetRocks doesnt even mention that buying a modern version of Delphi is an option companies should consider. And if they absolutely have to migrate to C#, then Elements is what professionals use.

On the whole, It’s alarming that the guys at .NetRocks seem clueless to the tools professional developers use for specialized work. Porting a Delphi codebase by hand its, well, amateurish at best. They might as well do a podcast on how to use ms-paint to create art, because Adobe Photoshop or Illustrator is not what people use (that was satire btw).

I am also pleased however, because .NetRocks unwittingly demonstrated my point to perfection 🙂 Instead of learning more about Anders Hejlsberg’s journey and life before he joined Microsoft, how the .Net class hierarchy is heavily influenced by the Delphi VCL framework, or how the concepts that ended up as .Net were in fact prototyped and experimented with earlier — they jumped straight to ridicule. Of a satire post (lol!).

You can click here for my full reply. And seriously, how it’s possible to mistake this old post as anything but heavy satire is beyond me. But I am thankful to .NetRocks for demonstrating my point. I could not have asked for a better example of arrogance and ignorance 🙂 Rock on!


 

13507225_10209662785317165_99874043994628045_n

When in Rome

Those that follow my blog or know me personally – also know that I don’t go out of my way to be confrontational or disrespectful. I try my best to follow the middle-path, to see positive in all things. But sometimes you face a level of stupid that begs a response. A verbal one. And this is one of those cases.

Lately I see more and more Delphi developers getting into debates with C# developers, and they are confronted with an attitude and belief-system that quite frankly is utter nonsense. It’s not based on history or facts but disputes rooted in childish “isms”. A mix of old habits and an unhealthy obsession with the notion that new means better.

It is nothing short of an intellectual emergency.

Old is bad?

If that is the case then C/C++ should be utter shit, because C is 3 years older than Pascal. Pascal was created to replace C and get away from (amongst other things) the absurd and time-consuming header file practice, not to mention the recursive hell that still occur when headers reference other headers and conditional symbols pile up.

Do you know how curly brackets became popular? Jump into your nearest Tardis and set the destination to the mid 1960’s. Back when mainframes were the size of Portugal and 1024 bytes of memory was the bomb. You see, back then memory was sparse and pretty much all the languages (or toolkits) needed to save space. So the { } curly’s were used as tokens for words like “begin” and “end”. Why? Because one ascii byte is less than five ascii bytes. Every byte counts when you only have a few Kb of memory.

Ps: when I use the word toolkit here I really mean a random soup of asm snippets on a loom. The bones collected to make C was actually a mix of homebrew toolkits made by different engineering teams at the time. And each of these toolkits had their own fanclub /slash/ legal representation; because all of them, miraculously, had invented the exact same thing at the same time before everyone else. These guys kept on fighting over who owned what well into the Watcom era. It’s probably the longest standing demonstration of a grown-up tantrum since the roman emperor caligula outlawed clothing.

Fast forward to 1970 and the amount of memory on mainframes had already doubled a few times just like Moore had predicted five years earlier. The C standard that Dennis Ritchie had painstakingly wrangled from the cold, greedy hands of grumpy old technicians was due for an upgrade. Tests were done by very dull, very serious men in grey clothing – and in their horror they discovered that human beings would interact with words and phrases faster than symbols and glyphs. So a word like “begin” would be picked up by the brain faster than {. It is just knockout stuff isn’t it.

You cannot praise Anders as a genius architect and at the same time throw his life work in the thrash. Perhaps you should ask yourself why Anders loved object pascal so much that he dedicated half his career making sure universities had access to it.

Living on the edge with a red tie, Nicolaus Wirth rocks the scene!

Living on the edge, Nicolaus Wirth rocks the scene!

Enter Niklaus Wirth, a man so clever and academic that he doesn’t catch exceptions; He is the exception. But next to his colleagues over at Berkley our dear Niklaus is the proverbial rock-star. He decided that in order to make programming more accessible for students, easier to learn and above all, establishing a standard for safer code, that they should take science to heart and include that in a programming language design. So he put on his gravest lab coat and began to implement “a better C” where he applied the research that words are picked up by the brain faster than tokens and symbols. Nine months later pascal was born, and to celebrate Niklaus used colored socks that day. A risky bit of business that could have cost him his seat at the strangely brown sorority club, in which he was master of ceremonies. But hey, Nikolaus is a rebel and this was the swinging 70’s after all.

My point? This is why pascal uses “begin” and “end” as opposed to { and }. It’s one of the many aspects of the Pascal language that makes it so easy to master.

Whimsical satire aside: with the knowledge that C is actually older than pascal, not to mention that it is identical in features, depth and complexity to Pascal– what is it going to be? Do you still cling to the notion that older have to be of less value than new? Or could it in fact be that you haven’t really looked closer at object pascal to begin with? Have you even installed a modern version of Delphi and given it a worthy test drive? Or do you just speak out based on what you like and know, as opposed to what you don’t know and haven’t even tried.

My 8-year-old daughter is like that. She makes up her mind about food before she has even tasted it. Surely a trained engineer can do better?

And I say that because — if we follow your line of reasoning, C/C++ should be incapable of producing C#. How can something old and outdated possibly produce something new and exciting? You do know that C# is written in C and assembler right? And that Java and all the other high level languages out there didn’t spontaneously self assemble. They are all written in C, assembler or pascal. And there is a reason for that.

Just like nature have different levels, from particles to minerals, from minerals to bacteria, from bacteria to plants, from plants to insects (and so on) — so does a computer. Each of these levels have laws and principles that govern them, and no matter how much you want to introduce something from a higher level into a lower level – that’s just not how reality works.

foodchain

The fact that I have to explain this in 2016 demonstrates how education and academia has degraded after C and Pascal was replaced with Java.

The law is very simple: each level can only emit complexity upwards. So assembler is used to produce C and pascal, C and pascal is used to produce C++ and object pascal, C++ and object pascal is used to bring about every other single piece of tech you take for granted.

So when you criticize Delphi but use C#, you are just underlying that you dont really understand the difference between archetypical languages and context based languages.

It’s like saying Visual Basic 5 is cooler than assembler. You don’t debate with people like that, you just look at them with pity. I don’t know how many years a software engineering  degree is elsewhere, but if you just spent 3 years in college and 2 years at a university followed by one or two additional years getting your diploma – and this is the level of insight you graduate with then I pity you. And I pity your employer because sooner or later you will bring about the apocalypse.

If we apply rational logical thought to the proposition at hand, we cannot but conclude that the notion of “new is always better” is false. It is to mistake marketing for facts and capitalism for science. The fundamental principles of computing wont change because you favour a high-level programming language. The C/C++ embedded market wont magically roll over, because C# is a mitigated disaster on embedded devices. And I speak from experience, not preference.

Your teacher should have taught you this: “In a computer, like nature, the past is always alive“. Look closely at your brand new PC or Mac: Bios, Vesa screen modes, non linear memory, hardware interrupt vectors and a cpu that supports no less than 3 instruction sets. You have layer on layer of increasing complexity (a.k.a “the past is still there”). Why not call up Linus Torvalds and ask him why he’s not using C# in his line of work; or NVidia why their GPU pipeline only ships with C headers and not C# classes. I sure as hell wouldn’t want to be on the receiving end of that call.

The only C# compiler worth using, is actually RemObjects C# (the Elements compiler). Because unlike mono or ms’s compilers, Elements builds proper binaries on the same level as C/C++ and Pascal. You can actually write kernel modules with Elements if you like, something you cant do with Mono or Visual Studio. Heck, .net services still require a service host on Windows.

C# is better than Delphi?

Anders Hejlsberg, the father of many languages

Anders Hejlsberg, the father of many languages

Actually, it’s not. What today is known as .net, from its CIL intermediate language right down to the global assembly cache is pretty much Delphi with a new syntax parser on top. Yes you read that correctly the entire .net product family is practically Delphi refactored. The whole bytecode regime was one of the last skunkwork projects Anders worked on, and was actually described on Borland’s news-servers years before it magically re-appeared as Microsoft’s flagship product.

Anders Hejlsberg created and prototyped these technologies (as did many other companies, since Java made bytecodes cool) while he was working at Borland. And indeed, the father of C# and entire dot net stack is none other than the father of Delphi.

At Borland (in the borland.language newsfeed I vividly remember) these technologies went mentioned under the codename “portable Delphi”. I followed the thread on the news servers for quite some time. It was no secret that Borland was tinkering away on the next big thing, and I believe it was ultimately Borland’s response to Java. Because at the time Java had eaten quite a chunk of Borlands educational seat licenses. And had Anders continued working for Borland, Delphi and C++ builder would perhaps be bytecode based today.

Factoid: The first project Anders was assigned to when he joined Microsoft, was J#. Microsoft’s attempt at hijacking the Java language. Microsoft lost the legal battle and was forced to pivot. The end result would be C#.

So when you, the C# fanboy, propose that Delphi is somehow inferior to dot net, or that object pascal is outdated and technically behind C# you quite frankly have no clue what you are talking about. Is it generics? We have that. Is a rich and powerful RTTI? Got that covered. Anonymous procedures? Its there too. In fact, the whole misunderstanding here stems from the fact that C# developers generally think “Delphi 7” is what Delphi is all about. Which is the same that saying that .Net  is what came out in version 1. Who the hell uses Delphi 7 in our day and age.

But just to make this crystal clear: Anders Hejlsberg is not just the father of C# and dot net: he is also the father of Delphi. And before that he gave birth to Turbo Pascal. In fact, Anders have 3 (if not four) Pascal development systems behind him before he came to Microsoft.

Borland, land of the free

Borland, the company Anders worked for, was put out to pasture by Microsoft. Bill Gates launched a financial onslaught that few companies on the planet could have endured. The reason was that Microsoft really had the worst developer tools on the market (Visual Basic and Visual C++), while Borland represented the very best. So for a long time Borland utterly demolished Microsoft when it came to software development.

When Borland later began to flirt with Linux (Delphi Kylix) it must have made the R&D department at Microsoft piss themselves. Delphi is responsible for thousands of desktop applications synonymous with Windows – and if Linux users got their hands on RAD tools like Delphi, with thousands of applications ready to be ported over? Do the math. You have to remember that this was before cloud computing. Microsoft didn’t have a fallback strategy and depended on desktop and server sales. Linux could do everything Windows could, but it lacked the desktop applications, the user friendliness and features Borland made simple.

Weird Science, a fun but completely fictional movie from 1985. I just posted this to liven up an otherwise boring trip down memory lane

Wishing for stuff doesn’t make it true, except in the movies

What Microsoft did was, basically, to buy out Anders (they also picked up a huge part of Borlands R&D department) from Borland and at the same time attack the company from every angle. Drowning them in bullshit lawsuits, patent claims and all the nasty, cowardly practices Microsoft was into back then (we all remember Netscape and how that went).

Bill Gates called Anders up personally and gave him “an offer I could not refuse“. Whatever that means.

So to put this myth to bed and be done with it: C# as a language and dot net as a technology did not originate with Microsoft. It has Borland written all over it and its architecture is intimately joined at the hip with Delphi and C++ builder. Companies like Microsoft dont innovate any more, they buy competence and re-brands it as their own.

Dot Net framework, you mean the VCL?

If that is not enough, take a long, hard look at how the .net framework is organized. Pay special attention to the features and methods of the base-classes and RTTI. What you are looking at is the VCL, the visual component library, Delphi and C++ builder’s run time library.

If you know Delphi and it’s RTL intimately as well as C#, you quickly recognize the architecture. The syntax might be different, but the organization and names are all too familiar.

There was never any secret where this came from, but it’s not advertised either so I don’t blame people for not recognizing it. But when people start to behave nasty over what should be either a passionate hobby or a professional discipline, then truth must be told. The very foundation of the dot net framework is taken straight out of the VCL.

And why shouldn’t it? The whole platform was whispered about under the codename “portable Delphi” to begin with, it was even designed by the same author – so why should it come as a surprise that it’s basically Delphi reincarnated with a different syntax parser? Either way a few bells should start ringing in your head – telling you that you may have been wrong about Delphi and object pascal in general. Probably because you know very little about Delphi to begin with.

So, what you perhaps believe sets C# apart from Delphi, what you imagine gives dot net an advantage over Delphi – is in reality the opposite: it is Delphi. It originated in Delphi, was inspired by Delphi and flows straight from the inventor of Delphi himself. Anders is responsible for two distinct object pascal development platforms prior to working for Microsoft. Take some time to reflect on that.

You cannot praise Anders as a genius architect and at the same time throw his life work in the thrash. Perhaps you should ask yourself why Anders loved object pascal so much that he dedicated half his career making sure universities had access to it.

Strength in diversity

The fact that C/C++ is alive and well today serves to prove my point: age has no meaning when dealing with fundamental technology. These languages embody the fundamental principles of computing. They don’t grow old because they represent a fundamental level that can never be replaced. There is now law that says C must be there, what persists is ultimately the features that C brings – spanning from assembly to the desktop. Those same features can be found only in one other language, namely Pascal.

Things are the way they are for a reason. And programmers before you established standards that address things you might not have to deal with, but that’s because they dealt with it before you. Just because you don’t see the plumbing doesn’t mean it’s not there.

These are the compilers responsible for the foundation on which all other languages, operating-systems, applications and services rest. Remove those, and the whole house of cards come crashing down.

So C/C++ and object pascal continue to exist, thrive even, precisely because they interface with the hardware itself, not a virtual machine or bytecode format. Delphi as a product is naturally tailored for operating-systems, as is C/C++. But that is because we use programming languages to create applications. There is nothing in the way of someone using compilers like freepascal and writing an operative system from scratch.

Ultibo is an ARM operative system written purely in object pascal

Ultibo is an ARM operative system written purely in object pascal

By contrast, C#, Java and Visual Basic are context dependent languages. They exist purely in context with an operative system or run-time environment. They can’t exist without these and lack the depth C and pascal brings to the table. The depth to span many levels, from the lowest all the way up to cloud and domain driven programming.

Just look at Android, an operative system written in Java right? Well that’s not really true. The Android operative system depends on a native bootstrap, a piece of code that establishes the run-time environment for Java applications. So when you fire up your Android phone, tablet or set-top box, the first thing to boot is a loading mechanism written in vanilla C. This in turn mounts a filesystem and continues to loads in bunch of native drivers. Drivers for the display, storage, wi-fi and all the hardware components that ultimately make up your SoC (system on a chip).

Its only after this long process that the environment is capable of hosting the Java virtual machine. Then and only then is the device capable of executing Java. This is why languages like Java is called context-based, because they run in context with an established system.

7709-f-1

Notice the “C” part of the Java NDK? Wasnt that exactly what Java was supposed to solve? All that hype and here you are, back at where you started!

So Java, which for the past decade or more have bombarded us with advertising like “platform independence” is a total flop! Its like communism and looks good on paper, but once you start testing these ideas in real life – they just don’t work.

Turned out that the java virtual machine when running on small devices like phones and tablets, choked the CPU. Which in turn used more electricity to keep up, which in turn rendered the battery life of these devices null and void. Talk about irony. All that hype and Java couldn’t even deliver its most basic promise: platform independence. And you removed C and pascal from our universities for this? It’s a disaster en-par with the burning of the great library of Alexandria.

There is more than a drop of ego and arrogance in this material. Young people especially like to believe they are smarter than those who lived before them. It’s a very selfish idea that really need to be addressed. Take someone like Alan Turing for example, the inventor of what we call computers today. This man assembled and created the first computer by hand. He even hand carved the wheels that made the mechanism run (which was mechanical during the second world war). I would happily bet $1000 that Alan could out program the lot of us had he gotten the chance. He has been dead for many years, but to imagine yourself smarter or more capable than him would be the definition of arrogance.

PS: This is where we come into the realm of wisdom versus knowledge. The difference may be subtle, but it’s so important to distinguish between them.

C# would have been a sad little language without the work of this man, Miguel de icaza

C# would have been a sad little language without the work of this man, Miguel de icaza

Either way without the native context to support it Java turned out to be a blundering behemoth. And its the exact same with C#. This is why Miguel de Icaza work on Mono has been so instrumental to the success of dot net and C# in particular. It was Miguel and his team that pushed C# into native compilation versus JIT for mobile devices. C# would never have been allowed on IOS or Android otherwise.

So that is the difference between archetypical languages and context based languages. Hopefully you now see why this negative attitude towards Delphi and object pascal doesn’t make sense. It really is just as stupid as these cults that believe in dinosaurs and that human beings once lived side by side with a T-rex. Anyone with a highschool diploma should instinctively know better.

The problem is when people in great enough numbers start to believe in this nonsense, because then they can actually damage important mechanisms of our eco-system. Insane beliefs really is dangerous both socially and technically.

Grow up!

I have no problem using C# and consider it a valuable tool in my toolbox. Same with Delphi. But when it comes to kick-ass desktop applications for Windows and OS X, Delphi has no rival. It gives you a productive edge that I havent found in any other language. And I don’t base that on preference. I base that on engineering and the years of development invested in Delphi by both Borland and Embarcadero.

I just find it odd that C# developers seem to lack the ability to speak well about Delphi, because it’s a tell-tell sign of immaturity to be incapable of that. Object pascal and C++ is identical in every way (except multiple inheritance, which even C++ programmers shun like the plague so no loss there). Why then should educated programmers find it reasonable to praise C but thrash pascal? It’s like saying that one car is faster than another – but when you pop the hood they have the same engine. The exact same machine code generator even.  It is psychological masturbation. We are coders, we shouldnt do things like that. We should examine the system carefully and then make up our minds.

And what is a programming language anyway? It is the ability to describe the solution to a problem. The ability to define and control how a series of activities execute. Each language represents a methodology for solving problems. A mindset. Nothing more.

Languages like C, C++ and object pascal will always exist, they will always thrive, because they are instrumental to everything else (desktop, operative system, services, everything that makes a computer capable of what it does). It doesn’t mean that Delphi is the solution to all problems, far from it. And that is ok.

Context based languages are more often than not designed for a purpose. Java was initially designed as a network language. This makes Java infinitely more suitable for dealing with that line of work. You can do the exact same in C but you would have to add a few libraries and establish a platform. Which is exactly what C and pascal is good at, implementing fundamental technology. Object pascal is a language you would use when creating other languages. C is more popular for virtual machine development, no doubt there, but there is nothing in C/C++ that you wont find in object pascal.

Interestingly, languages that many believe is long since extinct, like Cobol, are actually alive. Cobol was created to deal with large stacks of numbers and to push associated data through formulas. That is its context and no other language is better suited for the banking sector than Cobol. Which incidentally is the one place where Cobol is used to this day. And why not? Why waste millions trying to hammer in a nail with a hacksaw? Why not use a language especially designed for that purpose? SQL is likewise a language especially tailored for its tasks. I don’t hear anyone bitching about how old and outdated that is. It wont grow old because it embodies the principle of relational data.

And just what exactly is old about Delphi? Generics? Posix support? Weak and strong references? Garbage collection? Attributes? class, record and enum helpers? Dependency injection? Model view controller frameworks? The ability to write assembler side by side with pascal? The ability to link with C/C++ objects? Cross compilation for Windows, OS X and mobile platforms? What manner of logic are you using where you praise other languages for these features, but for Delphi the same features makes it old? Stop repeating what someone once told you like a mindless parrot and investigate. You are an engineer, a field of science; start acting like one.

In my work I use several languages: JavaScript, Delphi, Smart Pascal, Mono C# and even Python. I have also fallen in love with LUA believe it or not.

For web-pages I prefer Smart pascal which compiles to JavaScript (node.js is fantastic). Then you have shell scripting, a touch of x86 assembly code, a dash of bash on my Linux box and .. well, there is strength in diversity.

It all depends on what is best for the customer and what language yields the most robust solution. Sometimes C# is better suited for my clients infrastructure, other times Delphi makes more sense. But if I was to pick a single language to cover all of them, it would either be object pascal or C++ (gcc). These really have no rivals and will never go away.

These are the compilers responsible for the foundation on which all other languages, operating-systems, applications and services rest. Remove those, and the whole house of cards come crashing down.

So using many languages is actually an enriching experience. It brings more depth to our way of thinking and (consequently) affect the quality of our products.

Anyways, I believe congratulations are in order, because if your favorite language is C#, you have in fact been using Delphi without knowing it. There is so much Delphi in C# and they both originate from the same author. So why not pay a visit to C#’s older brother?

So go download Delphi, Lazarus and Remobjects Oxygene – enjoy the decades of research and excellence these dialects represent. It won’t hurt you, it will probably do you a great deal of good.

Smart codecs, a draft

October 18, 2016 5 comments

When you hear the word codec, what comes to your mind? For me I get flashbacks to early 2000’s when I worked in visual C++ and Visual Basic on a huge media project. ActiveX was a total cluster fu**, with a proverbial vipers nest of interfaces, vague explanations and more blue-screens than i care to remember.

Working on the RTL for Smart Mobile Studio has forced me to learn a lot of topics that I would, had I chosen not to go down the JavaScript rabbit hole, never have given a second thought. And I have now reached the point where codecs is starting to make more and more sense for the RTL. Let me explain why.

Codec is just short for “encode decode”, and a codec represents just an isolated ability to encode or decode a piece of data. If you have followed the evolution of Smart Mobile Studio and know your way around the RTL – you must have noticed that we have a fair bit of duplicate code hanging around. So far I have found two wildly different CRC implementations, a sexy and fast RC4 encryption routine, RLE compression, URL encoding, Base64 encoding, UTF8 encoding .. the list goes on.

All of these routines, no matter how small or large they may be, does the exact same thing: they encode or decode a piece of data. So you input one type of data and you get something else back.

Part of our RTL update is not just about adding cool new features, it’s also about order, cleaning up and getting the RTL into a form that is better suited for the future. I have already blogged about namespaces and the important role they play in the next version of Smart Mobile Studio. Well, I think its time to put our encoding/decoding capabilities into order as well.

Smart codecs

The first challenge that comes to mind when you sit down to create a codec framework, is the data formats you have to deal with. I mean, what is the point of a codec class if its only capable of processing strings right? And since Smart has all these cool mediums to play with, like streams, buffers and untyped arrays,  we want each codec to support as many such input and output types as possible.

This is where things get’s complicated and the critique I dished out to Microsoft ages ago comes back to haunt me. Because I now see why their codec system had to be so elaborate.

I mean, lets say you have a URL codec. It exists to take strings and replace characters that could be filtered out by the locale on another pc – so all it does is to replace these characters with a %charcode representation. That way the text can be re-mapped on the other side safely.

For a codec designed to process this type of data, you would have to define an input channel that can deal with text – as well as an output channel of the same type. But what if you want the output as a stream or memory buffer? Suddenly input and output gates become more complex, more fluid and data-dependent.

IBinaryTransport to the rescue

In the last update I did something magical. So magical in fact that I was oblivious to the fact that that I had just solved the problem of data mitigation. It was a 2 second mistake that turned out to be a stroke of genius (which I can’t take credit for, because I had no idea I was being genius at the time).

IBinaryTransport is just a super simple interface that gives you info about how much data is available, a reading method and a writing method, and finally the position of the read/write cursor. It’s stupendously simple, yet solves so many problems:

  IBinaryTransport = Interface
    function  DataOffset: integer;
    function  DataGetSize: integer;
    function  DataRead(const Offset: integer; const ByteCount: integer): TByteArray;
    procedure DataWrite(const Offset: integer; const Bytes: TByteArray);
  end;

What I did was to add this interface to both TStream, TAllocation and TBinaryData. I figured it would be handy when calling ToStream() and similar helper functions. But it turned out that with this interface in place – I could throw out TStreamReader, TStreamWriter, TBufferReader, TBufferwriter – and just write a single reader/writer class. TStreamReader actually doesnt have a clue what a stream is, because it speaks to the target via IBinaryTransport.

Same with TReader and TWriter for TBinaryData (memory buffer). They don’t know anything about memory at all, they just know how to communicate with the target through IBinaryTransport.

So how does this affect codecs? It means we can throw out the older model completely. Instead of having to define a whole list of input/output channels, add support for dealing with each of them — all we have to do is make the input and output channels IBinaryTransport (!)

This reduces the basic codec class to this:

  TCustomCodec = class(TObject, ICodecBinding, ICodecProcess )
  private
    FBindings:  TCodecBindingList;
    FCodecInfo: TCodecInfo;
  protected
    /* IMPLEMENTS :: ICodecBinding */
    procedure RegisterBinding(const Binding: TCodecBinding);
    procedure UnRegisterBinding(const Binding: TCodecBinding);
  protected
    /* IMPLEMENTS :: ICodecBinding */
    procedure Encode(const Source: IBinaryTransport;
        const Target: IBinaryTransport); virtual; abstract;

    procedure Decode(const Source: IBinaryTransport;
        const Target: IBinaryTransport); virtual; abstract;
  protected
    function  MakeCodecInfo: TCodecInfo; virtual;
  public
    property Info: TCodecInfo read FCodecInfo;
    constructor Create;virtual;
    destructor Destroy;Override;
  end;

And the work you have to do implementing a new codec to this:

  TURLCodec = class(TCustomCodec)
  protected
    (* IMPLEMENTS :: ICodecProcess *)
    procedure Encode(const Source: IBinaryTransport;
        const Target: IBinaryTransport); override;
    procedure Decode(const Source: IBinaryTransport;
        const Target: IBinaryTransport); override;
  protected
    function MakeCodecInfo: TCodecInfo; override;
  end;

Using a codec

A codec should never be used directly, but via proxy. This gives the framework a chance to create instances on demand, and also makes sure you don’t compile in codecs you never use. A few basic codec classes will be registered when your application starts – but these have always been compiled with your Smart application so there is little or no difference with regards to size or speed.

The benefits however are great: all codecs register with a central manager, they also register their mime-type (the format they work on), making it possible for you to query the codec-manager if it supports a specific encoding/decoding mechanism.

And naturally, it gives us uniformity with regards to data processing.

Here is an example of using the codec-api to URL encode a string:

var
  LBinding: TCodecBinding;
  LList: TCodecList;
  LSource: TStream;
  LTarget: TStream;
  LReader: TReader;
  LWriter: TWriter;
begin

  if CodecManager.QueryByName('url', LList) then
  begin
    LSource := TMemoryStream.Create;
    try
      LTarget := TMemoryStream.Create;
      try

        LWriter := TWriter.Create(LSource as IBinaryTransport);
        try
          LWriter.Options := [woEmulateCursor, woFlexibleBoundaries];
          LWriter.write( TDataType.StringToBytes("This is a string ready for URL-encoding!") );
        finally
          LWriter.free;
        end;
        LSource.position := 0;

        LBinding := TCodecBinding.Create(LList[0]);
        try
          LBinding.Input := LSource as IBinaryTransport;
          LBinding.Output := LTarget as IBinaryTransport;
          LBinding.Encode();
        finally
          LBinding.free;
        end;

        LTarget.position := 0;
        LReader := TReader.Create(LTarget as IBinaryTransport);
        try
          var LBytes := LReader.Read(LTarget.Size);
          writeln( TDataType.bytesToString(LBytes) );
        finally
          LReader.free;
        end;

      finally
        LTarget.free;
      end;
    finally
      LSource.free;
    end;
  end;

The cool part here is that the data can be anything. The source can be a stream, a buffer, a memory allocation – even a socket (!) Same with the target. As long as the target object implements IBinaryTransport the codecs will do their bussiness regardless of medium.

codecs

The output is an URL encoded string, processed at byte level

URL, Base64, UTF8, UTF16 .. and encryption ciphers naturally .. all these processes can now be represented under a single, efficient and small framework. And yes, there will naturally be top-level functions for these, same as before. I don’t expect people to write all of this every time they need to URL encode a string.

The point here is depth, and that Smart Mobile Studio applications can now do more and more of what you would expect from a native compiled system.