Archive for January 16, 2017

Smart Pascal: Node.js by example

January 16, 2017 Leave a comment

nodeThe update for the Smart Mobile Studio RTL is nearing completion. We still have a little to go, but all in all we have cleaned up the VJL considerably and made a much better organization. We now have 3 namespaces (read: groups of units), namely: System, SmartCL and SmartNJ. The latter representing the Node.js aspect of our codebase. We also have a fourth namespace in the making, namely Embedded. Currently our embedded support is limited to Espruino – but I have begun working on the Arduino and Arduino mega codebase. When these are complete they will all be in the Embedded namespace.

Ok, node.js so where do I begin. Perhaps its best to explain a few fundamental keys to how the new RTL is organized so you understand just much Smart gives you over vanilla JavaScript.

I mentioned the system namespace above, but what exactly does that mean? The system namespace contains universal code. Code that runs on all JavaScript platforms regardless of a DOM being present or not. So you get to enjoy stuff like:

  • Traditional TStream, TMemoryStream and TFileStream
  • Codec classes
  • Memory allocation and blue pointers
  • Encryption classes
  • Delayed execution (TW3Dispatch, system.time.pas)
  • Event objects
  • Filesystem
  • Datatype conversion (system.type.convert)
  • .. and the whole system namespace

Notice the TFileStream there? You will find that the system namspace contains an abstract implementation of a class called TFileSystem. And this is where things get funky. When you create a SmartCL application (that is, a visual HTML5 application), the implemented filesystem is in SmartCL.Filesystem. When you create a node.js application, the implemented filesystem can be found in SmartNJ.filesystem. See what I’m getting at here? The idea is that regardless of the runtime environment, you will always be able to use the same code no matter what engine you run on.

Ok, hopefully that gave you some context to work with, now let’s move on and create a fancy server for node.js !

Sexy server time

We are going to create a simple yet powerful UDP client / server application. I picked this because it’s sort of the odd-ball in our collection of server objects. UDP is connection-less and is ridiculously easy to use. At the same time there is no verification and assurance of delivery (that’s the flipside). But UDP is excellent for “live” logging, or for inter-service communication between micro-services on the same network or router.

Right, let’s just jump straight in:

var Server: TNJUDPServer;
Server := TNJUDPServer.Create;

Server.OnMessage := procedure (Sender: TObject; Data: variant;
  RequestInfo: TNJUDPRequestInfo)
  writeln("Recieved data:" + string(data));

Server.OnError := procedure (Sender: TObject;
  ErrorObj: TJsErrorObject)
  writelnF("Error[%s]: %s", [variant(ErrorObj.stack).toString(), ErrorObj.message]);

Server.OnClose := procedure (Sender: TObject; ErrorObj: TJsErrorObject)
  writeln("Server closed");

Server.OnAfterServerStarted := procedure (sender: TObject)
  server.Send("Your first Smart server is now running");

Server.Port := 1881;
Server.Address := '';
Server.MulticastLoopback := true;
Server.Broadcast := true;
server.Exclusive:= false;

Congratulations! You have just created your first node.js powered UDP server! It doesnt get much easier than this does it? Try to guess how much JS code you would have to write to get all the perks of the system namespace + the node.js namespace.

Some guy commented on an article I wrote the other day, asking me to give him a single reason why Smart Mobile Studio is better than just writing JavaScript. The answere is simple: producivity. Some of the code you get in a simple class in Smart represents hundreds or thousands of lines of JavaScript code.

What do you think is more productive? Maintaining 2 megabyte of raw JavaScript code; a binary kebab of spaghetti code that can go belly up if you missplace as much as a comma. Or the 2k of clean, easy to read, easy to understand object oriented pascal code?

I rest my case.

Right, with the server done already, let’s have a peek at the client:

LClient := TNJUDPClient.Create;
LClient.Port := 1881;
LClient.Address := "localhost";
LClient.OnAfterStarted := procedure (sender: TObject)
  LClient.Send("Client is now active", 1881, 'localhost',
  procedure (ErrorObj:   TJsErrorObject)
    writeln("An error occured:" + ErrorObj.Stack.ToString());
LClient.Active := true;

And that my friend is pretty much it!

Node.js really is awesome once you get to approach it on our terms. When you come to the JavaScript virtual machine through the OOP layer that Smart Pascal gives you, it’s a whole different reality.

Write a cluster ready, platform independent client/server application in less time than it takes to boot up Visual studio

Write a cluster ready, platform independent client/server application in less time than it takes to boot up Visual studio

What is the benefit here you might ask? All of this can be done in Delphi and Lazarus without much problem. The benefit is this: the generated code is platform independent, it will run on any platform as long as node.js is installed. It will behave identical regardless of operating-system. And you can cluster, clone and replicate instances to your heart’s desire.

Node.js is also easy to host, cheap and accessible. Native hosting is expensive and requires much more work. When something goes wrong inside a 200 megabyte service hosted on Amazon.. I have been there. Having to track down the bugs, re-build the executable while your boss is going mental – spend 10 minutes just uploading the damn thing, then you have to uninstall the service, reboot the whole instance, re-install the new .exe file and hope that you did manage to catch that bug.

Compare that to fixing the bug, uploading the new JS file and then inform PM2 that you need to hot-swap the service code. It’s a whole new paradigm.

Websocket server

With UDP behind us, let’s look at something a lot more complex. And when I write complex, I dont mean for you. Websocket is the fastest growing protocol standard these days. It’s basically an extension of the ordinary HTTP protocol. It is designed for long-term connections (read: not stateless, single shot operations like http) and is full duplex and async. So the server can talk to the client and visa versa without having to wait it’s turn. Just fire off as many messages as you like, whenever you like – and it will arive in the same order on the other side.

LServer: TNJWebSocketServer;

LServer := TNJWebSocketServer.Create;
LServer.ClientTracking := true;
LServer.Port := 1881;

// Setup our own protocol commands
LServer.On("command1", procedure (const Socket: JWsSocket; const Data: variant)
    writeln("Command #1 executed, recieved data:");

LServer.On("command2", procedure (const Socket: JWsSocket; const Data: variant)
    writeln("Command #2 executed");
    // broadcast a "hello" response with a byte array to *ALL*
    // connected clients. Making chat applications is very easy here
    socket.Emit("Hello", [12,13,14,14]);

LServer.Active := true;

If you look closely you will notice the use of the On keyword. This is what we use to define our server-side protocol. It’s essentially trigger words that you associate with a piece of code. When the server recieves a message it will read the message-name, look up a associated code and execute it. But let’s look at the background for this so you dont end up downloading the wrong package.

Normally you install the node.js websocket package first, and then you install something called on top of that. The On keyword we used above is actually not a part of the default websocket standard (which is more low level). This functionality is provided by a separate package called

But, since most people install and use websocket just to use, it makes sense to merge these two packages into a single, stand-alone distro that gives you everything in one go. And this package is cleverly named Websocket-IO (it’s just knockout names isnt it).

I usually install node packages in my project output folder

I usually install node packages in my project output folder

If you want to read more about websocket-io, head over to the npm website and have a peek at:

Client: Needless to say, the client class is just as simple so I dont really see a point in repeating that. In fact, all servers and clients inherit from the same base-class.

What should be stressed is that websocket is easy to use from Delphi as well. So if you have a native client and want to talk with your node.js server, I strongly suggest you stay clear of REST (which is one of the most wasteful protocols ever invented) and instead use websocket.

Note: since websocket is an extension to http, it is perfectly safe to use in a commercial environment. REST requires a lot more CPU and generates much more data on the network than websocket. The most time consuming and intensive aspects of a http call is during connection, and websocket is based on async and full duplex communication, with message caching and more handled automatically – so there is no reason why you should chose REST over websocket.

Websocket allows you to broadcast messages, or to filter messages based on criteria. Messaging and data-flow protocols are very easy to implement

Websocket allows you to broadcast messages, or to filter messages based on criteria. Messaging and data-flow protocols are very easy to implement

When you add the fact that websocket is also allowed from the browser, you can offer your services to the Javascript community without any extra development. So you get to cater for both native and JS clients. Being able to connect and interact with your custom servers directly from the browser opens up new possebilities. Want to display live information? With a full duplex connection the server can now inform you whenever something has changed instead of your clients polling on interval.

A Delphi extension to websocket (for Indy) was implemented and released by Andre Mussche a while back. The code is super easy to work with and bridges the world of Smart Pascal and Delphi quite nicely:

If you need information about how to use websocket regardless of language, just google the topic. There are thousands of resources out there.

Last but not least

I hope you have found this super simple introduction to node.js server coding inspirational. Node.js is a huge topic and there are roughly 350.000 different packages available online for it. Writing wrapper classes for node packages is not hard either. You do have to know your way around object pascal and Javascript, but if you look at my units and how we have solved it, you should pick it up very fast.

In the future I hope to generate a 1:1 import tool that will download, convert and install packages for you automatically.

Cheers guys!