Smart Pascal, what’s next?

For those of you who have been following my developer’s log over the years, both here (and on www.smartmobilestudio.com), I have a tradition to write a few words about where we are heading next after each release. So with the release of version 2.2 here is a peek of the future and what we will be cooking up in the lab next. Just for you!

The secret lab where all the magic happens

As a team we have worked hard for the past years, actually I started on it back in 2010, Eric joined in early, somewhere between 2010 and 2011 – at the same time as Jørn E.  Angeltveit (owner and CEO of Optimale Systemer AS, the original publishing company). And together with solid developers like Primoz Gabrielcic, Christian Budde and Andre Mussche, we formed the OP4JS consortium in 2012. Dedicated to a new object pascal for a new age.

That was 4 years ago. And today Smart Pascal is not only a distinct dialect of object pascal, it is also a licensed, stock based company: The Smart Company AS. Standing on its own two feet and continuing to grow.

We have seen Smart Mobile Studio (SMS) go from being an “Adobe Flash” like programming environment, into a fully fledged IDE with a decent form designer, support for packages (yes you can create packages and register components just like Lazarus and Delphi).

But is that all it’s ever going to be? Hardly. And I know people have a lot of questions about where we are heading, so I figured it’s time to shed some light on the subject.

Platform independence

 

In our recent release we have given our customers a whole year worth of research and development. I know people expected more frequent updates, perhaps 4 updates which was our initial goal – but had we done so, the RTL would have seen incomplete for 3 of those, so we decided to wait until the foundation was solid and in place.

The NodeJS service editor will make its debug very soon

To demonstrate: take something simple; Let’s take TMemoryStream. Easy right? Its been with Delphi since day 1, its one of the first things you learn about in DIY programming books, and considering we live in 2016 streams is something you take for granted.

Well guess what, that could not be further from the truth when it comes to JavaScript. The wonderful language itself may hold 51% of the global computing markedshare, but like the author himself has stated on numerous occasions: it was slapped together in record time and was nowhere near complete when Netscape pushed it. So streams, strong type checking, datatype conversion, sandboxed memory management and all the stuff we would expect a modern language to support — it just isn’t there. It has slowly appeared in various unfriendly forms (and Typescript naturally, which imitates aspects of Smart Pascal, cheeky Anders Hejlsberg), but to overcome these shortcomings we have had to implement even the most fundamental functionality from scratch, in Smart Pascal itself.

Something as trivial as converting a 32 bit integer into 4 bytes, or turning a TDateTime (64 bit float) into an 8 byte array is something JavaScript developers have struggled with for quite some time (using bit-shift is the typical solution, we use a faster method). They have been reduced to cheap hacks, like packing bytes as character data; the proverbial assault on the heap.

So before we could even begin to introduce stuff like TStream, we first had to research and implement consistent behavior between  platforms. What platform you may ask? Doesnt Smart Pascal target normal browsers? Absolutely, but there is a difference between browser, engine and renderer. And we also have much larger ambitions than just browser apps.

Browsers render, engines execute

Browsers may look the same but beneath the hood they can be as different as Delphi is from Pearl or Python. The list of browsers we want to support is long and growing, separated by builds (age of mobile device, type of device and even PC graphics card capabilities):

• There is Mozilla Firefox
• Safari, extended version of webkit
• Chrome, standard webkit
• Internet Explorer
• Microsoft Spartan browser
• Opera
• Epiphany
• Midori, customized webkit

And those are just for desktop PC’s or embedded systems (like the Raspberry PI) running Windows , OS X or Linux in some form or another. Six of these browsers are further separated by desktop and mobile editions where differences can be monumental (like CSS gradients suddenly altering direction and syntax between CEF2 and CEF3).

Technically speaking each browser and engine (where engine means the JavaScript execution module, like Webkit V8, Mozilla Spidermonkey or IO.JS) have both subtle and radical differences which we have to care about. Tiny but important details like timing issues, synchronization differences, priorities for dispatching and performance considerations.

Now throw NodeJS and IO.JS (initially a fork of NodeJS which has gone in a slightly new direction) into the mix; combine with embedded SoC running JavaScript natively (more and more hardware support JS as their primary automation language) and you get the idea of just how much our work is involved. Things like converting integers to bytes and back again seem trivial, child like even, but they are of utmost importance. Something as simple as using UInt8ClampedArray instead of UInt8Array on the wrong engine, browser or build can kill your application before it even starts.

So streams, just to get back to that, involves a little more than just slapping together a class. To get streams working we first had to build the foundation for streams to exist in the first place:

• Replicate object pascal strict types
• Research fastest and most efficient way of doing binary conversion
• Implement native type conversion in JavaScript itself
• Implement real memory handling:
  • The ability to allocate memory (TAllocation)
  • The ability to modify memory (TBinaryData)
• Differentiate between various encoding standards
• Implement marshaled pointers (offset references)

Take a moment to reflect on how much research, testing, prototyping and solution finding these simple concepts represents. Things Delphi has had over 21 years to reach, evolve and perfect.

And before Delphi became the product we all use and love today, it first went through 4 year of evolution as “Turbo Pascal for Windows”, releasing a grand total of 4 editions before that amalgamated into Delphi. And before that, seven years of Turbo Pascal earning Borland millions of dollars.

Engine and platform standard library

With version 2.2 just out the door, you get to enjoy elegant solutions to things JavaScript developers struggle with to this very day. Streams, memory management, writing custom controls with full inheritance, composite layout — all with rich support for NodeJS and various mobile and embedded platforms. But we are not done, not by a long-shot.

Writing object pascal services using standard RTL classes, running on affordable NodeJS hosting servers – from small vendors to giants like Microsoft Azure and Amazon cloud service is the next big goal.

NodeJS and IO.JS may look like plain old JavaScript but they are worlds apart from the environment you find in a browser. We are already hard at work separating the RTL into 3 distinct namespaces. This complete picture will appear later, most likely in version 2.2.5 or even 2.3. And with it you will enjoy many of the top-level benefits Delphi and Lazarus have in their native codebase. But this time, you can run it anywhere, even on your TV which no doubt have a browser!

• Unified standard classes between namespaces
• Node and IO Service application type
  • Command server
  • WebSocket server
  • DB server (and drivers)
• Unified DB API
• Non visual components
• Assets management (pre-loading of resources for easy distribution)
• TActions for visual elements
• IDE Embedded development

Embedded development

This is being worked on by a third-party and will be exciting for IOT programmers using the Raspberry PI (for example). A version of ARM Ubuntu is being tailored to communicate with our IDE. The Linux image will run NodeJS services you write in Smart Mobile Studio and can also (as a bonus) render a visual front-end application directly to the framebuffer.

With the internet of things (IOT) you need a language that connects

In other words: you will have full control over the entire device, and can use SMS as the tool to create single-application IOT devices. Examples of products you can create with this technology are Apple TV or Plex devices, ChromeStick, NetFlix, kiosk systems; Development from A-Z will can be done directly inSmart Mobile Studio. The IDE will also communicate with the IOT device, uploading your compiled NodeJS services, databases, assets and visual project UI code.

The Smart Desktop runs on Raspberry PI on boot, with NodeJS dealing with system level operations in the background. So much fun to work with!

While I love all things Smart Pascal, this is by far my favorite. IOT is important and allows you to create just about everything you can imagine.

And the beauty? It’s all done using standard protocols. Your NodeJS service can be hosted on Azure or Amazon, or just run quietly on your Raspberry PI or Beaglebone. It makes no difference. The code and classes will be exactly the same.

You are going to love it.

 

Smart Pascal record arrays, brute force search

Having introduced the “self-less” class to my readers in an earlier post (the buddha class as I call it), I started thinking about the practical sides of this. Some may just go “oh, so you have an object with no self, so what?”). Well its actually a big deal, especially when talking with servers and dealing with the myriad of non-standard data formats out there.

Array of record

Dictionaries are great because, behind the scenes, they are organized in such a way that the key you use to read or write a value, can be more or less instantaniously verified, looked up and accessed.

Dictionaries can be made in a number of ways, using various formulas for turning a string-key into a meaningful number (which in turn points to the data it represents).

Finding stuff in unstructured data is fun

But this article is not about fancy dictionaries, but rather something resembling an array of unstructured record. Where each item in the array can actually differ from the other both in content and layout  (!) Just think about it: under JavaScript libraries throw different structures around like candy, and there is no such thing as a fixed standard.

So let’s say you call some NodeJS webservice and it nukes you back with 100, 10000 or 1000000 records with completely different structure and content — that would pose quite a challenge using ordinary Delphi or FreePascal (well, not difficult but time consuming at least).

Let’s see just how fast we can push JavaScript to search a large mass of unstructured data. Note: I will be using code which is not yet in the RTL here, just so you dont freak out. The Ticks functionality will be introduced in the next Smart Update.

Right, so let’s start with the basics, lets give things some names:

type
TW3Associate  = variant;
TW3Associates = variant;

Variant under Smart Pascal is not the same as a Delphi, C++ or C# variant. It is just a way of saying “we dont know the type” of something. It essentially maps to JObject which is the most basic, empty “object” JavaScript can muster.

Next, Let’s create some data to play with:

procedure TForm1.W3Button2Click(Sender: TObject);
var
  Associates: Array of variant;
begin

  for var x:=1 to 1000000 do
  begin
    Associates.add(
      class
      cssname := "Item#" + IntToStr(x);
      typevalue := x;
      end
    );
  end;

end;

This gives us an array of one million JavaScript objects, each with two properties. Next step is to use some old school, brute-force searching. We are going to use loop-expansion, meaning that instead of a simple for/next loop dealing with a single item at a time – we are going to process 8 items per cycle.

function SearchAssociates(const Field:String;const Value:String;
  const Associates:TW3Associates):TW3Associate;
begin
  result := unassigned;

  (* Setup look expansion *)
  var index    := 0;
  var LLongs   := associates.length shr 3;
  var LSingles := associates.length mod 8;

  (* Search batch of 8 items *)
  while (LLongs > 0) do
  begin
    if Associates[index][Field] = Value then
    begin result := Associates[index]; exit; end;
    inc(index);

    if Associates[index][Field] = Value then
    begin result := Associates[index]; exit; end;
    inc(index);

    if Associates[index][Field] = Value then
    begin result := Associates[index]; exit; end;
    inc(index);

    if Associates[index][Field] = Value then
    begin result := Associates[index]; exit; end;
    inc(index);

    if Associates[index][Field] = Value then
    begin result := Associates[index]; exit; end;
    inc(index);

    if Associates[index][Field] = Value then
    begin result := Associates[index]; exit; end;
    inc(index);

    if Associates[index][Field] = Value then
    begin result := Associates[index]; exit; end;
    inc(index);

    if Associates[index][Field] = Value then
    begin result := Associates[index]; exit; end;
    inc(index);

    dec(LLongs);
  end;

  (* Search odd single items at the end *)
  while (LSingles > 0) do
  begin
    if Associates[index][Field] = Value then
    begin result := Associates[index]; exit; end;
    inc(index);
    dec(LSingles);
  end;
end;

And then let’s see how fast it can chew through these “untyped” arrays. And please remember that from JavaScripts’s point of view, each of these structures are unique. It cannot, like Delphi, take for granted that the next object is equal to the previous.

The test code now looks like:

procedure TForm1.W3Button2Click(Sender: TObject);
var
  Associates: Array of variant;
begin

  for var x:=1 to 1000000 do
  begin
    Associates.add(
      class
      cssname := "Item#" + IntToStr(x);
      typevalue := x;
      end
    );
  end;

  writeln('Performing brute force search:');

  var StartTime: Integer;
  var StopTime: Integer;

  StartTime := TW3Dispatch.Ticks;
  var data := SearchAssociates("cssname", "Item#999976",Variant(Associates));
  StopTime := TW3Dispatch.Ticks;

  if (data) then
  begin
    writeln("Found object:");
    writeln(Data);
  end else
  writeln("Search failed!");

  writeln("Ticks spent:" + (StopTime-StartTime).toString );
end;

Results

I ran the test 10 times, but only the first five produced different values – the engine then settled on 22016.

• 40960
• 22016
• 20992
• 20992
• 24960

Ticks is calculated as the number of ms since 01.01.1970. There are 621355968000000000 ticks from Ist Jan 1900
to Ist Jan 1970 (hence the const). And 10000 ticks represents a millisecond:

const
  CNT_TICKS_BASE = 621355968000000000;

class function TW3Dispatch.Ticks:Integer;
begin
  result := new JDate().getTime() * 1000 + CNT_TICKS_BASE;
end;

Conclusion?

Old school pascal beats both ForEach() and Filter(), methods introduced into JavaScript to deal with this kind of scenario. Goes to show, object pascal may be old (C++ is 3 years older actually), but it’s a damn fine language and makes NodeJS and web programming fun!

Smart Pascal and the Buddha object

I’m not sure how to introduce the full implications of this topic. It greatly depends on your background with curly languages (C#, C++, Java) and your love for all things Delphi, FreePascal, Oxygene and Smart Mobile.

But stick with me, it will be fun!

The buddha object

In traditional object pascal all classes must inherit from TObject right? No matter what you do you just could not escape that fact. Unless you used records, defined functions as fields and mapped the entry points manually. Which we sometimes have to do when talking to old-school C libraries or esoteric kernel code, but that’s another matter. TObject is the mother of all classes, thats the general rule (or was).

But what if someone took the time to completely re-design this from scratch. I mean the whole inheritance system inside the compiler? What if we got rid of TObject as the ultimate ancestor and instead introduced a “self-less” object? Kind of like a Buddha object? One that can become anything?

Live prototypes

Our chief compiler engineer, Eric Grange, is nothing short of a wizard when it comes to these things. So a while back he introduced exactly what I described above. So now we get to do super cool tricks that C#, Java and JavaScript have been doing for a while.

Passing self-less classes as parameters and calling back into it!

So what just happened above? Well, when the user clicks a button (w3button5 in this case), we call the method “ProcedureToCall”. But wait! We pass as the parameter a class! A class of no specific type, which requires no initiation (no constructor).

But notice how we set a new value (UnKnown.NewValue) which was not in the original declaration. And after that, we call back into the ghost object and dump the “this” property.

Without getting to technical, “this” means the same as “self”. But since this is an anonymous ghost class, it has no self — so “this” then refers to the current context. Now let’s look at what the output is when we dump the object to console:

Its all there, even the new property

Pretty cool huh? Now let’s look at the code generated for these procedures. I think you will agree it’s both readable and straight forward:

That’s almost a 1:1 code generation!

Mind: Blown!

If you don’t quite get why this is important and powerful, then think about what this does for talking with external libraries, refactoring older code, and most importantly – constructing objects in real-time by code. Once you start playing around with this I think you will agree — it’s the bomb!

Now ponder why W3Button5 has a “self” parameter 😉