And storage for all, Smart solutions to HTML5 files

There is a reason we have sort of left the storage mechanisms for the browser alone. The event-driven nature of JavaScript makes reading and writing of data into a nightmare. Just to store a file you have to respond to 5 (five!) nested events. So forgive me if I havent had time to do something about this until now – but trust me when I say that LocalStorage and SessionStorage which came with the RTL is one of the best solutions you can find.

In short, it goes like this:

• You have to ask to store data into your quota
• You have to then ask to get access to the filesystem
• You have to ask for access to a file-object
• You must request a filereader or writer
• An operation may not execute straight away

Each of these steps happen inside the event handler of the previous.The tricky part here is that these are all nested, which means you can use each element only within *that* event handler. So if you try to access the filesystem object outside the event-handler where you ask for a quota change you will actually get an error; or the JS engine starts to behave oddly. I had to re-boot Chrome because it really crashed when I tried to trick it into behaving linear like we are used to under Delphi and C#.

In Smart it would look something like this:

RequestQuota(Procedure ()
  begin
    requestFileSystem( procedure ()
    begin
      requestFileObject( procedure ()
      begin
        requestWriter( procedure ()
        begin
          writedata( procedure ()
          begin
            showmessage("Data written! Yeeej!");
          end);
        end);
      end);
    end);
  end);

In other words — it’s a complete mess. It’s actually twice as bad as above, because you also have to define error handlers besides success handlers. Wish we could get R2/D2’s soon so we dont have to deal with code like this.

File actions

Anyways, since storing files (and reading them) is such a recursive mess, I could probably leave the topic alone and get away with it. But I’m not going to do that. In typical “me” style (which some hate or love, or both) I jump head-first into the mess and get to work, even though this has to be one of the worst tasks I have dealt with in a long time. And that should speak volumes of how absurd the JavaScript file-system is organized.

To make a long story short, the best way to deal with files is through actions. Thats right, the same glorious action system that you find in Delphi and FreePascal. In essence an action is just an object with two common methods: validate() and execute(). The validate function should return the readiness of the object (are all properties set? Can we perform the action now?), while the execute() method implements the actual activity.

I hated this implementation when I first made it, but it was the only system which would secure that your data goes from A to B safely. Initially I wanted a unified storage API where drivers was created after an analysis of the browser environment. So if you were running under phonegap, you would get a driver with full access to the filesystem. If you were running in a normal browser, then you would get a driver which saved data to local storage.

That was a good idea, but sadly the recursive callback loops could not be overcome to provide a linear, blocking storage API like we are used to under Delphi (or any native language for that matter).

The good news

The good news in all of this is that Actions perform very well! I mean, you tend to isolate reading and writing of data in one place, so creating an action object is no different from creating a file-object. And it’s the same with loading. Under HTML5 loading data on a mobile device typically involve a dialog with a spinner or something while you load your data — so triggering this from an action is easier than through 2-3 event handlers.

Secondly, with the streaming system in place you dont have to bend the knee to JavaScript. You write the data as you always have with streams, streamwriters and true binary memory. And when you are done composing your file-format, just execute a pre-defined action for saving and forget about it (how cool is that!).

So how do we store files? Like this:

var
  mAction:  TFileActionSave;
begin
  mAction:=TFileActionSave.Create;
  mAction.FileName:="document.txt";
  mAction.FileMode:=fmPermanent;
  mAction.FileData:=FStreamToSave;
  mAction.Execute(NIL);

Pretty neat huh? And if you want to be notified when the file is properly written to disk (or whatever storage medium you select), then you can just assign an event handler, like this:

var
  mAction:  TFileActionSave;
begin
  mAction:=TFileActionSave.Create;
  mAction.FileName:="document.txt";
  mAction.FileMode:=fmPermanent;
  mAction.FileData:=FStreamToSave;
  mAction.OnExecute:=procedure (sender:Tobject)
    begin
      showmessage("File has been saved!");
      TFileActionSave(sender).free;
    end;
  mAction.Execute(NIL);

Device separated actions

Actions for performing singular activities are not new, Delphi has had them for ages. And they are really helpful (especially for UI programming). But Delphi’s actions are a bit different from our type of actions here. Our type of actions are expected to be executed once, then disposed or left to be eaten by the garbage collector. You can recycle them naturally but it serves little purpose.

In the event handler above we dispose of the action in the OnExecute() handler, just to make sure no reference to the filesystem lingers in the system. You don’t have to do this, but it’s always best to write things properly. It costs little and can make a huge difference. Some browsers dont take well to having a filesystem object floating around in memory. Chrome goes bananas if more than one reference targets the same object for instance.

Either way, the really cool part is that you can now write your IO code once, and it will run on all supported platforms. If you execute this after phonegap compiling your app, it will store files on your device’s real filesystem. If you execute it in a normal browser (like you are using now) then it will store data in the sandboxed environment (of which you have a maximum of 5 megabytes to play with).

The final result

Well, having cleaned up this mess as best I can, here is the final result. I have made the example below a bit messy on purpose just to show you how simple actions are.

What this routine does is to first generate a small file, execute the “save” action — and when the save routine is done and the file is stored – we create a loading action and read it back again.

It will write out the string to the console. And if you dont believe this is the simple version, feel free to check out system.io.pas when we release this code later. I think you will be very thankful that you have Smart Mobile Studio to deal with all this so you can focus on the fun stuff – writing cool applications!

procedure TForm1.W3Button4Click(Sender: TObject);
var
  mAction:  TFileActionSave;
begin
  mAction:=TFileActionSave.Create;
  mAction.FileName:="document.txt";
  mAction.FileMode:=fmTemporary;
  mAction.OnExecute:=procedure (sender:Tobject)
    begin
      var mLoader:=TFileActionLoad.Create;
      mLoader.FileData:=TMemoryStream.Create;
      mLoader.FileName:='document.txt';
      mLoader.FileMode:=fmTemporary;
      mLoader.OnExecute:=Procedure (sender:TObject)
        begin
          try
            var mReader:=TReader.Create(mLoader.FileData);
            writeln(mReader.ReadString);
          except
            on e: exception do
            writeln(e.message);
          end;
        end;
      mLoader.Execute;
    end;
  mAction.FileData:=TMemoryStream.Create;
  var mTemp:=TWriter.Create(mAction.FileData);
  mTemp.WriteString("This is some data");
  mAction.Execute;
end;

Well – enjoy!

Smart Mobile Studio, workers by code tip

Web workers work for you!

Webworkers are essentially the browsers answer to classical threads. Although I cant for the life of me understand why the W3C have gone to such lengths to utterly screw up their standard, I can do my best to rectify their mess with easy Smart functions 🙂

In short, here is the challenge: Web workers must be stored in a separate file. So you can’t tell the browser to just take an object or method and “use that as a thread”. I wish you could do that, it would make my life easier – but sadly you can’t. You are expected to have separately compiled JavaScript worker-files, which know nothing about the main application. That last part is actually quite OK, since you don’t want multiple threads messing up your GUI.

The second challenge is that you can’t even share variables. All communication between a worker and your main application must be performed via messaging. So a worker can only receive and send messages. To start some “work” you essentially send over a text-based command (use JSON to stringify records for instance) telling the worker what to do.

Creating workers by code

Turns out there are legal ways of bypassing the first Webworker rules, namely that of a separate file. You can write your script to a blob (binary large object) and then use the URL composer to generate a URL for your object. This will be a unique resource locator which the browser can use to load data from. In short, here is how you do it:

function makeWebWorker(Script:String):THandle;
var
  URL:    Variant;
  BLOB:   Variant;
  WORKER: Variant;
  mTemp:  Variant;
begin
  asm
    @URL    = window.URL || window.webkitURL;
    @BLOB   = window.Blob || window.webkitBlob;
    @WORKER = window.Worker || window.webkitWorker;
    var @mTemp = new @BLOB([@Script]);
    @result = new @WORKER(( @URL).createObjectURL(@mTemp));
  end;
end;

You can now create as many workers as you like by code, like this:

// create the worker
var FWorker:=makeWebWorker(
  #"onMessage: function (msg) {
    postmessage(msg);
   }");

// setup message handler for when the worker sends us messages
FWorker.onMessage := procedure (msg:string)
 begin
  writeln("Message back from worker!");
  writeln(msg);
 end;

// send message to worker
FWorker.postmessage("Hello this is a test!");

And now you are probably thinking — oh why can’t i write Smart Pascal worker code?
Well there are several reasons for that. First of all, since the worker is 100% isolated it means that it wont recognize the RTL. The RTL would have to be loaded into the worker in order to function properly. And even then there are problems, because the VMT would be different between the main program and the worker program.

However, a completely clean, non RTL dependent piece of smart pascal will work! But the only way to get that done is to isolate it in a separate file.

But the trick above is handy, especially if all you want to do is check for features without interrupting the main program. And if you know JS fairly well you can also write some serious message-handling in pure JS which can be reflected by objects on the other side.

Another example

Below if a fully-blown example which successfully establishes a worker, installs a message-handler on both ends, and sends a message through. And it’s all done from the same source-code. Please note that getting the source of a function does not work on class members, only stand-alone functions.

Enjoy the code!

function makeWebWorker(Script:String):THandle;
var
  URL:    Variant;
  BLOB:   Variant;
  WORKER: Variant;
  mTemp:  Variant;
begin
  asm
    @URL    = window.URL || window.webkitURL;
    @BLOB   = window.Blob || window.webkitBlob;
    @WORKER = window.Worker || window.webkitWorker;
    var @mTemp = new @BLOB([@Script]);
    @result = new @WORKER(( @URL).createObjectURL(@mTemp));
  end;
end;

procedure TestCode;
var
  mContext: THandle;
begin
  asm
    @mContext = this;
  end;
  mContext.onmessage := procedure (msg:variant)
    begin
      asm
      console.log(@msg);
      end;
    end;
end;

procedure TForm1.W3Button3Click(Sender: TObject);
var
  mRef: TProcedureRef;
  mCode:  String;
begin
  mRef:=@testcode;
 asm
  @mCode = @mref;
 end;
 mCode += "TestCode();";
 var mWorker := makeWebWorker(mCode);
 mWorker.onmessage:=Procedure (msg:variant)
  begin
    writeln("We got a message back:" + String(msg.data));
  end;
 mWorker.onerror:=procedure (msg:Variant)
 begin
  showmessage("An error occured!");
 end;
 mWorker.postMessage("Hello dude!");
end;

Getting data into your Smart Mobile apps

The new reality is all about small, powerful mobile devices. The 80’s was all about home computing, the 90’s about creativity and the 2k’s about connectivity. Well we have mastered all of those aspects of computing, and now it’s the age of mobility! Not just for devices you put in your pocket, but for the data that comes with it.

Getting data

No matter if you are writing your next iPhone or Android application with Delphi + FMX or Smart Mobile Studio + Phonegap, your data has to come from somewhere. Unless it’s a very small, very limited notepad application where your customer is expected to populate it from top to bottom; but that would be a very strange app in our day and age.

Getting data from your company servers or perhaps the latest public service is not always easy. But with the advent of WebSockets this has pretty much changed – if you have paid attention to the web technology that is.

So what are WebSockets? In short it’s just like normal networking sockets, but with a few limitations. You are allowed to connect, send ordinary text and also binary data. The latter is in the form of blobs or untyped array buffers. Dont worry, we have taken care for all that for you, so sit back and relax and enjoy the code.

Also websockets are automatically created for long sessions, so the underlying architecture will try to keep the connection alive. This is very important since we dont want to re-connect over and over again just to send small packets. So it’s more economic to keep a single connection alive that both server and client read and write to.

Creating a server

First you need to download the WebSocket server (based on Indy which is installed with Delphi). This can be done from this website. If you havent written a server before, don’t worry – it’s actually quite fun! Especially with WebSocket since essentially what you do is read commands, fetch data, send a response – and then wait for another command again. But you should get a book on the subject if you are a complete newbie, or get acquainted with the concept by trying Indy from Delphi first.

Server design is also fun because you get to define the protocol your mobile devices should use! You can go for a simple text-based protocol, you can opt for superobject (JSON objects) based packets, XML or just invent something completely new. You may also want to stick to standards so that people can buy access to your services? Quite a few people make a living creating just web-services. It’s a rather lucrative market to be honest.

Note: I urge you to create your first server as a normal GUI program, and only later when everything is working 100% isolate it as a Windows Service or Linux Daemon. There are bound to be errors (network coding is no different from other types of coding) and it’s easier to debug and display those errors with a GUI.

Creating the client

Your apps liven up with data from a server

This is the fun part! Once you have some rudimentary server up and running, be it an echo service or a serious power-house of a database service, Smart Mobile Studio will help you deal with it.

Working with web-sockets is really not that different from using the ordinary Indy components. You call commands and handle events, like OnMessage which signals that a text has been received from the server. Remember that you can send and deal with binary data through Base64 encoding. JSON is perfect for stringifying complex datatypes – so everything is really for the picking here.

Below is a websocket class ready to be played with. Please note that this uses the new RTL classes, which means you have to be a part of the beta team to compile it. Although it will compile if you remove all the write() method except the string based one.

Either way, it demonstrates how easy it is to transport data between your company server – and your Smart Mobile Studio application. Websockets are super awesome, easy to use, they work on all platforms (and all popular browsers), on mobile devices just as desktop — so it’s the easiest route to get your data from A to B.

  TWebSocketHandle  = THandle;

  TWebSocketState = (stError,stIdle, stConnecting, stConnected, stClosing, stClosed);

  TWebSocketOpenEvent     = procedure (sender:TWebSocket);
  TWebSocketCloseEvent    = procedure (Sender:TWebSocket);
  TWebSocketErrorEvent    = procedure (Sender:TWebSocket);
  TWebSocketMessageEvent  = Procedure (Sender:TWebSocket;Value:String);

  EWebSocket = Class(EW3Exception);
  TWebSocket = Class(TObject)
  private
    FHandle:    TWebSocketHandle;
    FOnOpen:    TWebSocketOpenEvent;
    FOnClose:   TWebSocketCloseEvent;
    FOnMessage: TWebSocketMessageEvent;
    FOnError:   TWebSocketErrorEvent;
  public
    Property    OnOpen:TWebSocketOpenEvent read FOnOpen write FOnOpen;
    Property    OnClosed:TWebSocketCloseEvent read FOnClose write FOnClose;
    Property    OnMessage:TWebSocketMessageEvent
                read FOnmessage write FOnmessage;
    Property    OnError:TWebSocketErrorEvent
                read FOnError write FOnError;

    function    SocketState:TWebSocketState;
    Function    Connected:Boolean;
    function    URL:String;
    function    Protocol:String;

    Procedure   Connect(URL:String;Protocols:Array of String);
    Procedure   Write(value:String);overload;
    procedure   Write(Value:TMemoryHandle);Overload;
    Procedure   Write(Value:TStream);overload;
    Procedure   Write(Const Data:TBinaryData);overload;

    procedure   Disconnect;
    Destructor  Destroy;Override;
  end;

uses  W3C.DOM,
      W3C.TypedArray,
      W3C.WebSocket;

//############################################################################
// TWebSocket
//############################################################################

Destructor TWebSocket.Destroy;
Begin
  if (FHandle) then
  Disconnect;
  inherited;
end;

function TWebSocket.Protocol:String;
begin
  if (FHandle) then
  result:=JWebSocket(FHandle).protocol;
end;

function  TWebSocket.URL:String;
begin
  if (FHandle) then
  result:=JWebSocket(FHandle).url;
end;

function TWebSocket.SocketState:TWebSocketState;
	const
		CONNECTING: Integer = 0;
		OPEN: Integer = 1;
		CLOSING: Integer = 2;
		CLOSED: Integer = 3;
begin
  if (FHandle) then
  begin
    case JWebSocket(FHandle).readyState of
    CONNECTING: result:=stConnecting;
    OPEN:       result:=stConnected;
    CLOSING:    result:=stClosing;
    CLOSED:     result:=stClosed;
    else        result:=stError;
    end;
  end else
  result:=stIdle;
end;

Function TWebSocket.Connected:Boolean;
begin
  result:=not (SocketState in [stIdle,stClosed,stError]);
end;

Procedure TWebSocket.Connect(URL:String;Protocols:Array of String);
begin
  (* disconnect socket if already connected *)
  if connected then
  disconnect;

  (* Allocate new socket *)
  try
    asm
    (@self.FHandle) = new WebSocket(@url,@protocols);
    end;

    JWebSocket(FHandle).onclose:=Procedure ()
      begin
        if assigned(FOnClose) then
        FOnClose(self);
      end;

    JWebSocket(FHandle).onopen:=Procedure ()
      Begin
        if assigned(FOnopen) then
        FOnOpen(self);
      end;

    JWebSocket(FHandle).onmessage := procedure ()
    var
      event:  Variant;
    begin
      asm
        @event = event;
      end;
      if assigned(FOnMessage) then
      FOnMessage(self,String(event.data));
    end;

    JWebSocket(FHandle).onerror := procedure ()
    begin
      if assigned(FOnError) then
      FOnError(self);
    end;

  except
    on e: exception do
    Raise EWebSocket.CreateFmt
    ('Connect failed, system thew exception %s [%s]',[e.classname,e.message]);
  end;

end;

procedure TWebSocket.Disconnect;
begin
  if Connected then
  begin
    try
      try
        JWebSocket(FHandle).close();
      except
        on e: exception do;
      end;
    finally
      FHandle:=NULL;
    end;
  end;
end;

Procedure TWebSocket.Write(value:String);
begin
  JWebSocket(FHandle).send(value);
end;

procedure TWebSocket.Write(Value:TMemoryHandle);
begin
  JWebSocket(Fhandle).send(JArrayBufferView(Value));
end;

Procedure TWebSocket.Write(Value:TStream);
var
  mRaw: TMemoryHandle;
begin
  if Value<>NIL then
  begin
    if Value.Size>0 then
    Begin
      Value.Position:=0;
      var mBytes:=Value.Read(Value.Size);
      mRaw:=TDataType.BytesToTypedArray(mBytes);
      Write(mRaw);
    end;
  end;
end;

Procedure TWebSocket.Write(Const Data:TBinaryData);
begin
  if Data<>NIL then
  Begin
    if Data.Size>0 then
    Write(Data.ToTypedArray);
  end;
end;

SQLite + Smart Mobile Studio = True

In case you missed the great moment, here is a link to my previous post on the subject.

In short: Someone has ported the engine SQLite database engine to JavaScript. This means that you can now create, query, save, load and otherwise do whatever you wish without relying on the browser to provide you with a native DB API. This means that if you desire SQL database support in your mobile applications then Smart Mobile Studio now provides that out of the box!

I must mention though that we also provide our own lightweight TW3Dataset class, which for known table structures is much more efficient and faster to work with. But TW3Dataset does not have SQL support and searching for records must be done “the old way”. The SQLite engine applies internal indexes to make searching for data fast and simple, and you get to use SQL which can simplify complex expressions.

Libraries

The SQLite implementation is pure JavaScript and will be stored in the libraries folder. When you include the sqlite unit in your projects, this library is automatically included – and the file copied to your /res/ folder.

To initialize the SQLite driver you simply do like this:

procedure TForm1.InitializeForm;
begin
  inherited;
  w3Button1.enabled:=False;
  w3_RequestAnimationFrame( procedure ()
    begin
      SyncDBStart;
    end);
end;

Procedure TForm1.SyncDBStart;
begin
  if CheckDBEngine then
  begin
    FDB:=TSQLiteDatabase.Create;
    w3button1.enabled:=True;
  end else
  w3_RequestAnimationFrame(SyncDBStart);
end;

As you can see from this code, I disable the GUI while the database engine is loading. I use an async-loop to check if the driver has initialized, and only when true do I re-enable the GUI and create an instance of the database.

You may want to modify your application unit and make sure the driver is loaded before you show a form, or perhaps show a fancy CSS3 dialog with “Loading, please wait” until all your resources are in place? The new system.fileutils.pas unit should give you some inspiration there 😉

Excellent performance and great storage use

Well, here is a picture of the testbed app. Notice the bytesize of a SQLite database with 10.000 inserted records (!)
This means that you can easily stuff thousands of records into a database stored in LocalStorage (max 5 megabytes)! Perfect for lazy updates where you upload the data to your server when the device is connected to the internet.

Enjoy!

Smart Mobile Studio and the SQLite engine (!)

SQLite is now 100% native JS

I am so excited right now I can hardly type straight. This is so awesome that I hardly know where to begin!

Some guy decided to port over the whole SQLite engine from C to JavaScript.  That’s right, not an interface to use the DLL, or read the tables or anything like that; a complete and full port of the entire SQLite database engine converted to JavaScript.

This means that you dont need to fiddle with WebSQL or whatever restriction browser vendors have put on reading data. The entire SQLite engine itself is now capable of running inside your Smart Mobile Studio applications.

So you can load, download and even generate binary SQLite database files. If you have a script on your server to accept uploads you can even save those files — or what about creating a SQLite editor so people can design their tables in the browser?

Needless to say I am really happy about this find and jumped straight into it to write a wrapper. I’m not sure we will be able to get this into the next update – but if not, you can download it from this website. It is just to fantastic not to include.

This means you don’t get one DB engine in the next update – but two! system.dataset and system.sqlite all in one go.

Who would have thought – SQLite converted to JavaScript running happily in the browser

Smart Mobile Studio, customize your new IDE

The smart mobile studio designer has never really been any good. We wanted to use a commercial designer at one point, but due to technical limitations in the components we were unable to do so. In short, having more than one instance of a designer in the application caused some serious conflicts and instability.

Instead we decided to stick with our own variation, most of us never really used the designer because are used to FreePascal, C# and library coding under Delphi. The designer for iOS in XCode on the mac is hopeless, we we tend to create even the design in our code. So this decision didn’t have such a hard impact on us. Sadly that was not the case for our users who wanted to design HTML5 interfaces without the clutter of Dreamweaver or dullness of Web-Matrix.

So I can only say I’m sorry we had to do this, but the code generation and support for hardware was more important at the time.

But times change, and right now it’s the tools that needs attention! All in all you will be happy to learn that in the next update which is just around the corner, the designer and all it’s features has received a lot of love!

The Delphi style designer in action

First, don’t let my insanely blue color theme in the picture above bother you, I was just playing around with the palette. You are free to either disable theme support or pick another which suits your taste better. The same goes for the text-editor colors. The IDE now ships with several preset-files, and you can create your own palette which can be saved, loaded and made default with a single click.

But back to the designer, what is really exciting is that we actually have 3 different designer modes (!) These are:

• iOS
• Delphi
• Wireframe

In the picture above we are using the Delphi inspired visual designer. It’s a bit dull but I must admit that for me this is the most familiar to use; and holy cow is it fast compared to our previous layout control! Christian has really outdone himself this time and created a designer we can all be proud of.

The first designer was my concoction, I never got to finish it and while it worked quite well with ordinary components, it was ill suited for threaded rendering which is required by the webkit engine. So Christian pretty much had to throw everything out and write it from scratch using Graphics32 (which he has helped create).

For those interested, take a second to contemplate just what takes place inside our designer at any given point. Because it’s not “simple” X/Y coding:

As you create your layout we actually build a small program of your layout, where all the components are created and positioned out. This tiny program is then compiled in a separate thread, and ultimately rendered by the WebKit engine to an off-screen pixmap. We then talk to JavaScript directly from Delphi and enum the position (including rotated and/or scaled offsets) for each control -before we grab the control pixels and cache them in buffers associated with each designer-element (phew!).

In short, this is how you can move and render HTML5 controls around inside a Delphi/FPC program just like they were ordinary VCL/FCL controls. Pretty neat right?

Either way, whatever our problems with the designer were in the past, that is now ancient history! The new form-designer is silky smooth, pixel perfect, easy to use and very, very fast!

Themes, palette and your taste in bling

Another really fantastic collection of features is how the new IDE deals with colors. We all love them and we all have different perception of what good UI design is like.

Well finally we have added the same amount of (if not more) options as Microsoft Visual Studio — from being able to customize the code editor, the gutter section and the mini-map; to applying a full theme to the IDE in general. Below is a picture of the IDE using a different theme from the image above.

Personalize the IDE just the way you want it

My personal favorites are “Amethyst Kamri “and and “Smokey Quartz Kamri”. Some love them and others hate them. But at least you have options to change the general look and feel of the IDE! And if you hate skinning and tweaking colors, just pick the default Windows theme and load in a bog-standard Lazarus/Delphi editor preset.

When we combine all this tweaking freedom with the completely overhauled and completely dockable IDE sections, you can pretty much personalize around 90% of the application.

A forgotten feature

The class browser was a great idea borrowed from the C# Mono project, but sadly we had to down-prioritize it earlier on – and it has remained in a sort of “limbo state” since version 1.2.

In the next update you will discover that we decided to give that old gem some love too and it really shows! Indexing is faster than ever, and searching for classes and methods is finally, after what seem like ages – efficient and useful!

The class browser was never created to be documentation. If you have some experience with C# or C++ through the mono project, you will know that a class-browser is just for searching up classnames/methods without having to change file or CTRL-Click out of your current unit.

C# programmers use this like mad, and assigns it to a clever key-combo. The .net framework is huge so remembering everything is impossible. Also, when focusing on a difficult task it’s not always productive having to scroll to the top of the unit, CTRL+Click on another unit, then CTRL+F to search (and so on). If all you want is the name of that class or a member –thats what the class browser is for. Nothing more, nothing less.

Finally it’s a breeze to hunt down a class to inspect it’s members

More to come

Over the past 2 weeks I have availed more and more features. And we are not done yet. There have been plenty of fixes and advancements in the compiler, linker and how applications are put together. So you have plenty to look forward to!

I have already mentioned quite a bit about the RTL changes, but not all — so stay tuned for more insight into what is without a doubt the best release of Smart Mobile Studio since 1.2 (!)

A better, Smarter Mobile Studio

It’s been a hectic couple of weeks. You know how it is before a project enters alpha mode. You want all your ducks in one row, you want no surprises and as much done so you can safely enter the beta phase. Well things are no different in our camp and tension can get high from time to time.

Today I am presenting the first screenshots of our “un skinned” Smart Mobile Studio IDE. This is the 2.5 candidate. While much may look more or less the same, hiding behind the UI are thousands of fixes, tweaks and improvements.

Quite a lot has changed, it’s hardly the same project any more

The biggest change, at least from a conceptual view, is that our UI is now fully dock-oriented. Meaning that you can drag almost everything around and position it where you like to. So if you prefer to keep the project manager on the right rather than the left, just drag it over and save the layout (you can also pick a default layout).

But there is more. Much, much more.

As you can see from the picture above, webkit-developer tools is made use of. This can be incredibly helpful when debugging. It also helps you familiarize yourself with exactly how our RTL operates, which creates HTML “on the fly” from JavaScript. A lot of people have problems understanding how this is possible, but once you get to play with it – everyone go “oh wow!”.

Dock galore! Setup the IDE just like you want it

The RTL is also getting some quality time; More than 50K lines of code has been added by me alone (not counting all the fixes and extras the others have added) so whatever you have missed from the RTL, chances are you will be happy about at least a couple of additions. I know I am ecstatic about the memory stuff, and stream! Oh god I have missed those!

Better construction rules

We have also added a few minor changes to how applications work internally. Now don’t get upset, it’s not a “code breaker” type of change. And if it should manage to break your code, it’s a 3 minute fix. But it’s worth it, because the benefits and speed are phenomenal.

Consider this: JavaScript is written from the ground up to be ASYNC. This means that when you create an element, like say a button or an image, there is no guarantee that the object is ready straight away.

In a 100% async framework there is no such thing as “right now”. So JavaScripts most sacred design-rule is that all types of work should be done in lazy mode. Perhaps the inventor was spanish or mexican – i dont know, but that’s what JS does. This is best seen in the constructor, because whatever you create there wont be visible and fully “ready” until after the constructor has exited.

Up until now we have ignored this, because the majority of our applications have been fast and small enough to not be bothered by this. But making large-scale applications means playing by the rules, so we decided to fix this sooner rather than later.

So all visual elements now has a new procedure: ObjectReady() which you should override and use to adjust and set object values. This method fires when the element itself and it’s children are 100% fully injected into the DOM.

So quick rule of thumb is:

• InitializeObject() is where child elements should be created
• ObjectReady() is where child elements are initialized with values
• Resize() is where we adjust and adapt to size changes
• Paint() is where we update graphics (only applies to TW3GraphicControl)
• PerformLayout() forces a manual resize and update of the UI

If you have created controls under Delphi this should be familiar steps. And to complete the picture we adopted the property “ComponentState” as used by Delphi and FreePascal; So now you can check and adapt to component states like you have always done:

if not (csReady in ComponentState) then
FLeft:=Value else
Begin
  BeginUpdate;
  FLeft:=Value;
  AddToComponentState([csMoved]);
  EndUpdate;
end;

Here are the ComponentState values:

• csCreating
• csLoading
• csReady
• csSized
• csMoved
• csDestroying

csCreating is only set while a control is being created. csLoading is only set while designer-code is executed. csReady is set and stays permanent once the construction sequence is complete. csSized and csMoved reflect movement and resize calls. These states are managed by the BeginUpdate and EndUpdate mechanism. And finally, csDestroying is set during destruction of the instance.

Packages and component installation

Although we added this in a previous update, we have spent some time polishing it, so in this version you can finally use packages exactly like you would under Delphi. In fact creating component packages is easier than under Delphi because you don’t have to fiddle with run-time and design-time paradigms. Simply include the units you have written (including resource files like images, sound or data), full-out the version info and required information fields – and that’s it. SMS will happily compile a package for you that you can sell, send to your colleagues or make available online.

Building packages is easier than under Delphi

Other tidbits

Loads. So many in fact that it should be divided into 3 updates rather than 1. Marshaled pointer support is probably the biggest game changer, combined with support for Move, FillChar and all the other “memory” methods. Built on top of this are buffers, steams and data conversion classes. This may sound simple and small, but believe you me – in the world of JavaScript this is unheard of.

We also updated the networking controls to reflect stream and buffer support, so now you can safely download data and get the binary result in a stream. Just like under Delphi or FPC.

More tools and time-saving features

And finally we have dataset support. w3system.dataset will finally bring you a uniform way of dealing with data. It’s an in-memory dataset written in Smart Pascal (read: no dependencies). This can be used to retrieve data from a nodeJS server or REST service; you can use it to transfer changes from a client to a server (offline editing). Only creativity sets the borders.

Well — I think you will be very happy with this update. A stable, rock solid RTL. A feature rich, tried and tested IDE. A compiler which is faster and produces better code than ever before — and fantastic support for all mobile platforms out there.

Smart Mobile Update, oh man this rocks!

Do you sometimes feel a bit envious of C# programmers with all their cool IDE features? Like automatic updates, built-in package manager which downloads, installs and does all the grunt work for you? I mean, Visual Studio’s nuget package manager looks almost like appstore, but with free packages which can be installed with a click.

I know I have. Being a Delphi/FPC programmer can be tiresome. Especially when it comes to component updates, RTL updates and god knows what else you have to manually deal with.

Well then It’s with great please I assume the role of whistleblower and show you a “peek” of the upcoming Smart Mobile Studio update architecture. That’s right, while you were sleeping we were building a server infrastructure – and one of the features is about to emerge publicly!

Oh look, it does everything for us!

Updates for all

Imagine this: Someone at HQ locates a bug in one of the RTL units. Today this means registering a case in fogbugz, providing reproducible code and much, much more. And depending on how critical it is – it can take a while before the ticket comes up to be fixed.

Our new system allows us to do immediate updates. So whenever we add a new feature or fix a bug, it’s fixed straight away — and it is available immediately. All you have to do is click “update” once in a while and you get all the changes, just like Visual Studio and Mono users do!

And we have channels, meaning different branches you can check out, like “Beta” and “Nightly builds” and so on. But please note that these are reserved for paying customers only. The update system takes care of license validation, tracking – updating your executables as well as the VJL + RTL codebase. So this is more or less a perfect solution for serious HTML5/Pascal programmers as you can get. And dont get me started on automatic package installation (ops! Did I say that out loud? Darn…).

I soooooo want to blow the whistle on our other features, but I can’t! Not yet. But the moment they are ready for public ears, rest assured that I’ll play the proverbial bagpipe on all of them 😀

Enjoy!

Microsoft Phone? Yes. And it rocks!

When I tried a Microsoft phone some eight years back it was a less than satisfactory experience. To call it a phone is perhaps to stretch the definition, it was more like a brick of HTC which could be used to make a call -or defend yourself against attacking marauders. No touch screen naturally, since that appeared a year or two later — and the only redeeming quality I can remember was the fact that it could be hooked-up to your PC and programmed quickly. I actually connected it, started visual studio and had a working “hello world” application in less than 10 minutes.

“I am actually tempted to get rid of my iPhone right now” 

While my general impression of that old Microsoft phone was terrible, the experience of “connect and code” has stayed with me. And my terrible experience had not so much to do with the OS, which was actually quite excellent at the time — but more with the physical phone itself.

So when I started coding apps for Apple a couple of years back- mentally I compared my experience to that of my first HTC/Microsoft experience. I was quite shocked to find that the Apple development process was so.. how can I put it, complete slavery? I mean having paid in blood to get a new Iphone, which is your property after all, you do expect to be able to put your own software on it right? Well not if Apple has anything to say about it. Seems to me that you don’t really own an Apple product, you lease it for the duration of its life-cycle. And you never get to decide what software you can run, especially not some home-brew code you cooked up yourself; No, you are allowed to pick from a pre-defined menu defined by Apple. A bit like politics where you are expected to select from a pre-defined list of candidates (pick your marionette of preference, but it makes no difference what so ever).

So while Apple may be the leading force in mobile technology today, I still remember Microsoft development as a much easier and pleasant experience. You did not have to digitally sign your own applications just to use them on your own hardware. Nor did you have to pay to create software for what is by definition your own property. And last but not least, creating and deploying in-house software to X number of devices in your organization was your own bussiness. Microsoft did not meddle in how you used your device, nor would that be a natural occuring thought. It’s your phone and what you put on that phone is your choice.

That was before Apple came along however.

Beware slumbering giants

Well with IPhone in hand today I went out and bought a Microsoft Lumia. It was so cheap that I simply could not say no. I need it for testing Smart Mobile Studio JavaScript code first and foremost, but I was also curious to see what a Microsoft phone could do; I mean compared to the gazillion Android offerings and onslaught of Apple quality assurance slogans we have learned to live with. That’s a pretty stiff competition by the way, so Microsoft have more than their work cut out for them.

Microsoft Lumia in all it’s squareness

Between you and me, I have a gut feeling that Microsoft is brewing on something. Have you noticed how Visual Studio suddenly have something called “Universal apps” lately? So you can create an app in C# that runs on Windows, XBox, Windows Embedded (read: tablets and drones) and Microsoft Phones.

This means that it’s going to be hard for developers to say “no” to Microsoft phone development since they get support for that free out of the box.

“Seriously. I had an Android phone for roughly six months and I ended up
giving it away with a 3000 NKR loss because I hated it so much”

So don’t be surprised if you start seeing more Microsoft phones out there — because while Microsoft has been slumbering, we all know that M$ is very, very good at one thing: making monopolies. With their complete abstraction from X86, we are going to see a tidal wave of cheap ARM devices flooding the market in 2-4 years running Windows 10.

What about the phone

Now to the phone! I love it! Seriously. I had an Android phone for roughly six months and I ended up giving it away with a 3000 NKR loss because I hated it so much (Samsung Galaxy S4). So forget Android is all I have to say.

Microsoft mobile is polished, intuitive and has everything you expect from a modern phone “and then sum”. Easy to call, easy to Skype, chat and perform all manners of modern communications. Stuff like E-mail and Facebook check-in is magically done for you once you have entered a couple of credentials, and you stay connected to your Microsoft account from the start. You can also sign up for an XBOX account if you want to download games or just testdrive some “universal apps” (see, told you they would arrive!). So getting the phone up and running was easy, polished and pleasant.

This is so cool!

In short: The phone is responsive, has great looking animated menus with a snappy interface. Wow, I cant believe I have ignored Microsoft for so many years, because this phone rocks!

Android never quite made the cut in my book. It’s to much of a mess with to many gaping security holes just waiting to be exploited (and they usually are). Apple is strung to tightly and bound on hands and feet to their money-making model which in the end serves only Apple and not the consumer.

Microsoft is the middle-ground, closer to Apple than to Android, but more polished and friendly than both of them combined. I absolutely have fallen in love with this device.

Oh and the battery life is exceptional! I have gotten used to my iPhone being constantly low on power. The Lumia was fully charged in 18 minutes and is not even at 70% 6 hours later (with constant use).

We need alternatives

The market desperately needs more than two operative systems. It’s a bit like politics. If you only have two parties you dont really have politics, just two factions putting on a show to dazzle the weak-minded masses who gladly relinquish their freedom for bread and circus. Which is exactly what we are seeing between Google Android and Apple iOS. They have some 300 (figuratively speaking) lawsuits against each other on patent infringement at any given point in time, while they throw more and more pointless features onto smaller and smaller footprints. It’s just plain silly how much crap we have gotten used to in such a short time. Remember when a phone was just a phone? Now it’s a fashion statement.

Android suffers from being open-source, with no legal entity taking true responsibility for quality (and forget Google, they like to have their name on it, but shun any responsibility). Although google is trying to up the quality of the Android store – let’s face it, the quality of apps on Android is ridicules compared to iOS. Apple may be security nazis with their application signing and serial key madness, but that at least has paid off with a huge catalog of quality titles.

Microsoft is nowhere near delivering the same number of apps (they have around 97.000 apps, Apple is probably near 500.000 by now). But I love this phone and I am actually tempted to get rid of my iPhone right now, just so I can buy a bigger and faster Microsoft phone. I picked up my Lumia for 780NKR which is peanuts compared to my iPhone 6 (8000 NKR). I can only imagine what type of device I could get from Microsoft at that price.

Well — I strongly urge you to try out the Microsoft phone before you get an Android device. Seriously, why would you want a phone that needs a virus killer? Android is a mess! It’s so-called “open policy” comes at a cost and that cost is the virus infected, scam ridden filth that is Android. A system which has allowed cheap corner-shops in asia to flood the market with fake low-quality devices fooling millions of people on a daily basis.

We need more alternatives, solid alternatives, something which is neither Android nor IOS. And believe it or not but Microsoft has nailed it with the Lumia. It’s one hell of a phone with all the cool stuff you expect from iOS, but with the freedom Android users brag about – but here it’s more mature, in the sense that all adults understand that there is no freedom without laws and boundaries.

Music for the masses

One special feature I want to mention is the free access to music Microsoft gives you. The phone allows you to create your own radio mixes. So you select a few bands you like, and based on that it creates a continuous playlist for you – including bands who fall into the same category. Once in a blue moon you get a few commercials – but this system is much easier to get started with than wimp. And it’s free (you can also sign up for premium services just like wimp).

What amazed me was that within 5 minutes of owning the phone, I was able to listen to a ton of free music. Compare that to Apple’s regime where every single item must be bought! Not to mention the unfair rules of Apple, where they simply ban competing products. Talk about unfair monopoly, Apple is far worse than Microsoft ever was if you ask me.

Also, the music features are built into the phone, And if you love music, you are going to love the easy access you get here.

Another thing is also the ring-tone creator. Again it’s free and comes with the phone. No more having to buy ring-tones, or fooling iTunes to use MP3 encoded files. Just copy over the mp3 you want or record something — cut, copy and paste. Done!

My new favorite

iOS has been my favorite so far due to quality of service and their well stocked app library, not to mention the ability to sync music and videos automatically. Which Android still wont do out of the box. You have to copy over MP3’s manually or buy an app and desktop app to help you. So in terms of technical excellence Android loses due to an operative system which seems to be put together by hob-goblins, lacking even the most rudimentary security. It also loses flat down in friendliness because -there must be a limit to the amount of settings a man has to tweak just to call his bloody girlfriend(!) It’s overkill, pure and simple – to present users with this level of variables. Some of them quite critical as well (affecting battery lifetime).

It’s just a silly trick to sell phones. The more “features” Android can list, the more impressed stupid people are who more often than not dont have a clue what it means. Does your mom benefit from controlling the level 2 Java cache? Ofcourse not! Are these really features? Let’s call a horse for a horse and see it for what it is –a sales pitch.

Well, whatever Apple or Google dazzle you with – Microsoft does it all, and they do it better by leaving out the rubbish and keeping the good stuff. In fact, the latest Microsoft phones are right up there with iOS, but with one massive advantage: It’s not bolted shut by nazis who insist of milking you for every single penny they can get. You are free to program apps running on your own phone, and you can even copy it over to other phones without asking for permission from the silicon papacy at Apple HQ to do so.

The moment you want to push your software out to the store you naturally have to go through a signing process, that’s to be expected. But the quality and ease is 100 times better than anything Apple is offering – so Microsoft wins on all accounts.

So I am happy to report that I have found my new favorite mobile device — and the source could not be more unexpected! It’s the Microsoft Lumia! So tomorrow im updating my Cell subscription and I’m going for the shiny new Lumia 930 — selling my IPhone in the process.

Good bye Apple, and thanks for all the fish!

Guess it’s time to whip out Smart Mobile Studio and Visual Studio and compile my first HTML5 App 🙂

Delegates, a fun look at multi-casting with Smart Pascal

This is just for fun, but here is the smallest possible delegate (read: multi-cast events) I could come up with. The lack of generics makes this tricky (its even tricky with generics), so this is probably as close as we get.

It should be easy enough to add operator overloading in the += and -= style made popular by Smart Pascal and C#, that way you could do stuff like:

// Install a new delegate handler
FHandle:=FDelegateOnClick += Procedure (sender:TObject;e:TDelegateParams)
  begin
  end;

// Uninstall handler
FDelegateOnClick -= FHandle;

Well, here is the code. Small and compact as always:

type

  TDelegateNameValuePair = Record
    nvName:   String;
    nvValue:  Variant;
  end;
  TDelegatePairList = array of TDelegateNameValuePair;

  TDelegateParams = class(TObject)
  private
    FData:    TDelegatePairList;
  public
    property  Data:TDelegatePairList read FData write FData;
    function  Set(const ValueName:String;
              const Data:Variant):TDelegateNameValuePair;
    function  Get(const ValueName:String):Variant;
  end;
  TDelegateParamsClass = Class of TDelegateParams;

  TDelegateProcedure  = procedure (sender:TObject;e:TDelegateParams);
  TDelegateEntryList = Array of TDelegateProcedure;

  TCustomDelegate = Class
  private
    FEntries: TDelegateEntryList;
  public
    function  CreateParams:TDelegateParams;virtual;
    function  Add(const Entrypoint:TDelegateProcedure):THandle;
    procedure Remove(Const Handle:THandle);
    procedure Invoke(const Params:Array of Variant);overload;virtual;
    procedure Invoke(const sender:TObject;const Params:TDelegateParams);overload;
  end;

//###########################################################################
// TDelegateParams
//###########################################################################

function TDelegateParams.Get(const ValueName:String):Variant;
var
  x:  integer;
begin
  result:=null;
  for x:=0 to FData.Count-1 do
  begin
    if sametext(ValueName,FData[x].nvName) then
    begin
      result:=FData[x].nvValue;
      break;
    end;
  end;
end;

function TDelegateParams.Set(const ValueName:String;
         const Data:Variant):TDelegateNameValuePair;
begin
  result.nvName:=ValueName;
  result.nvValue:=Data;
  FData.add(result);
end;

//###########################################################################
// TCustomDelegate
//###########################################################################

function TCustomDelegate.CreateParams:TDelegateParams;
begin
  result:=TDelegateParams.Create;
end;

procedure TCustomDelegate.Remove(Const Handle:THandle);
var
  x:  Integer;
  src:  THandle;
begin
  for x:=0 to FEntries.count-1 do
  begin
    asm @src = (@self).FEntries[@x]; end;
    if Src = Handle then
    begin
      FEntries.Delete(x,1);
      break;
    end;
  end;
end;

function TCustomDelegate.Add(const Entrypoint:TDelegateProcedure):THandle;
begin
  asm ((@self).FEntries).push(@Entrypoint); end;
  asm @result = @Entrypoint; end;
end;

procedure TCustomDelegate.Invoke(const sender:TObject;
          const Params:TDelegateParams);
var
  x:  integer;
begin
  for x:=0 to FEntries.count-1 do
  begin
    try
      FEntries[x](sender,Params);
    except
      on e: exception do;
    end;
  end;
end;

procedure TCustomDelegate.Invoke(const Params:Array of Variant);
var
  x,y:    integer;
  mParams:TDelegateParams;
begin
  for y:=0 to FEntries.count-1 do
  begin
    mParams:=createParams;
    for x:=0 to Params.count-1 do
    mParams.set("value" + x.toString,params[x]);
    try
      FEntries[y](self,mParams);
    except
      on e: exception do;
    end;
  end;
end;

Using the delegate

Since I did not include operator overloading, we resort to anonymous procedures to register our handler. Please note that TCustomDelegate.Add() return a reference-handle which you can use to un-install the handler procedure later (!)

Here is how you can go about it. Also notice that Invoke() is overloaded, so you can invoke the delegate passing the parameters as an array of variant.

var
  mDelegate:  TCustomDelegate;
begin

  mDelegate:=TCustomDelegate.Create;
  mDelegate.Add(
    Procedure (sender:TObject;e:TDelegateParams)
    begin
      showmessage(e.data[0].nvValue);
    end
  );
  mDelegate.Invoke([Variant("this is a test")]);

Remember to derive your own delegate objects, and also to override the CreateParams method to return your own parameter instances.

So in short: Inherit from the customdelegate with your own class. Override and derive your own TDelegateParams. You may also want to get rid of the read/write stuff since that is quite slow. Change that with your own parameters for the delegate.

This is actually how we create events and delegates in C# — so it’s hard work 🙂

Smart Mobile Studio, behind the scenes part 2

I’m going to give away a few more tidbits of behind the scenes as you wait for the next update. Let me assure you all that it’s on track (actually ahead of schedule) and I think everyone will feel we are putting in place “the missing pieces” of the RTL.

And being honest, It’s not easy knowing exactly what the missing pieces are, I mean, I spend just as much time in C# and C++ that I do in Delphi, and the C culture is almost fully source-code orientated. Meaning that programmers expect to write much more code than the average Delphi or FreePascal programmer. The tools and frameworks are likewise source-code only (with some exceptions), and every programmer is expected to tailor, extend and even write from-scratch controls. In the world of C you don’t have the luxury of drag & drop components. Although C# is more in the spirit of Delphi, but most component packages are source-only, only rarely do you find packages which installs into the component palette (bought commercial components tend to come that way, but very rarely open-source packages).

C# may look easy but it’s a lot harder than Object Pascal. And a lot less visual

So in some way I am surprised that so few people have picked up how fundamental it is to write your own custom controls with Smart. Under Delphi and FPC it can be quite a challenge to write advanced controls, but the Smart RTL is completely different and simplifies everything — once you understand how it generates HTML elements for you and how styles work.

So when we launched SMS we kinda of took it for granted that our customers would check out the competition and realize that Smart Mobile Studio was right up there with the best. Had you taken the time to look at MonoTouch from Xamarin for instance, besides being a C# framework, you would realize that it’s a much more demanding toolkit. You would have to override and extend the application object straight away for instance, so no lurking about with drag & drop designers (although they do market it much like Delphi was 10-15 years back).

In short: writing good code takes time, and there is no quick-fix to quality products, no matter how friendly your IDE is.

The good, the bad and the ugly

Right. We have noticed that quite a large chunk of our customer-base don’t really care too much about mobile applications. Which sounds odd considering it’s a mobile studio. Either way these customers can be categorized in two groups:

• Those that want to create a full website using Smart Mobile Studio
• Those who use Smart as a Flash replacement, writing banners, slideshows and interactive media

The demand for controls which fit into the whole “desktop” browser genre is growing. And yes, we have noticed your needs and we will not ignore you.

In fact, one of the controls that is getting a makeover is the scrollbars. These doesn’t really belong under iOS or Android platforms, but works very well in a desktop browser. Sadly the default CSS style I once made was butt ugly, and it was suffering from a “off by 1” bug. So time to pimp it up and get rid of the bug!

As always you can write some CSS for this yourself (and you are expected to do so, not just rely on our default schemas), but at least we now ship with something that resembles a normal scrollbar. But yes, just edit the stylesheet yourself and make it look different. CSS is easy and you will learn it in less than 1 hour.

Ta-Da: The new scrollbar look is simple, but at least you see what they are!

I have also taken the liberty to put some glyphs on the min/max buttons. These are hardcoded into the class, but you can remove or replace them easily (just override the InitializeObject method and change the innerHTML property).

Notice that they are transparent in the picture above? Well, this is actually by design. Just assign a color or background picture through the background property, and adjust them to your liking. A simple:

w3verticalscrollbar1.background.fromColor(clWhite);

Should do the trick. But I have left them transparent by default because you are intended to adapt these to your form and CSS style. Except for the scrollbox control, there I have styled them with a plain white background.

New widgets?

A couple of new controls have been added. First there is a new label component which is simpler than the one shipping with Smart Mobile Studio from version 1.x. The default label is actually a composite label, consisting of a child control to handle horizontal alignment. This is great for most situations, but I found myself missing a simpler label when I coded the CaseBook demo.

A plain, un-decorated iOS button has been added. This is a pure duplicate of the default iOS button. A white, rounded button with a blue verdana caption and blue frame. Simple but important.

We now have full support for iScroll

The next control is more exciting perhaps, namely a design surface control. This inherits from TW3Scrollbox, but introduces more functionality for building editors. So if you want to make a simple paint program (or a full photoshop clone), or perhaps a tile map editor for games, or a designer control which is better than ours– then this is the control you want to inherit from.

Connected to the above control are 2 controls which are interesting, namely horizontal and vertical rulers. This may not sound that hard to make (nor was it ) but they are tricky to get right. What I did was render the whole ruler in one go onto a TBitmap; assign it to the control as it’s background picture, and then I update the background picture position (phew) according to the position. A very neat and clever way of solving the redraw problem. And as a bonus it will trigger the GPU on mobile devices to make it snappy!

And last but not least, we now support the iScroll library. If you have no idea what this is then I suggest you google it. It’s a great library for doing iPhone type scrolling lists (which bounces when you reach the limit, has accelerated speed support and much more). I have implemented a new TW3Scrollbox based on this library, which is perfect for displaying text, menu options and news. I used this in the CaseBook demo.

More controls, and the cloud thing too!

Yes, more is on the way! I am presently going over MonoTouch and the default widgets from Apple to see what we can add. Smart Mobile Studio is first and foremost a programming environment for mobile platforms, so that comes first. But I guess it’s time to start expanding into HTML5 full frame applications (read: browser applications for desktop), so you can expect more in the future updates.

A treeview and listview control is on my list, as well as a proper database grid and string-grid.

Cloud storage, coming to a smart RTL near you in the future

But before we venture into map controls, grids and whatnot — I have an urge to solve a more immediate problem. Namely storage. TMS software has just released their Cloud package, and I really want to provide something similar for Smart Mobile Studio. Being able to store data directly in the cloud, per application user, is something which would set us apart from other HTML5 frameworks. At least with the simplicity and elegance I have in mind.

But it goes a bit deeper than just “being able to”. I want to amalgam cloud services with local storage to provide a unified storage mechanism; which includes full filesystem emulation.

Meaning that you would write code to store files and data once, and it would work the same no matter what the target is. Cloud, localstorage or sessionstorage – makes no difference.

Anything else?

Much more than I can fit here! I have talked over and over about the memory stuff, which is turning heads even for pure JS developers (which have not really seen real streams and encoding buffers like this before), so that is probably the biggest “wow” factor in the next update.

But let us not forget all the fixes and additions to the IDE! But for that you have to wait a bit more.. but rest assured, they are awesome!

The Smart Mobile Studio Competition!

I want to remind every one about the Smart Mobile Studio competition I launched earlier.

It’s still early (I have received two really good entries already), so if you own Smart Mobile Studio – or even if you don’t (heck you can download the trial version and have a go) and fancy winning the latest Raspberry PI 2 (model B, the new one that runs Windows 10) then this is your chance!

Probably the coolest mini-pc in the world

You can find the details regarding the compo here. The rules are straight forward and easy, and remember: this is not a graphical demonstration competition. This competition is about making the best component! And writing components is very entertaining, especially for a programming environment as young as Smart Mobile Studio. It lacks many of the controls which we take for granted under Lazarus or Delphi (hint).

It’s ready to be sent to your place

We are practically giving this Raspberry PI away; It’s just lying there in its box ready to be shipped to wherever you live; dying to be turned into something fun and useful. And even if you don’t need it, your kids will love playing every nintendo, amiga, sega and arcade game ever created (just install PIMame and turn it into an arcade machine).

So what are you waiting for? Impress us with a new toolbar, a new button-bank, sinus scrolltext or whatever control type you love to code! A hex editor would be nice (or any editor to be honest, I havent gotten around to coding one yet).

Smart Pascal Database, finally

Work is progressing nicely on the Smart Mobile Studio update. The whole team is busy adding new features, fixing bug reports and testing code. It’s a wonderful sight to see more and more of our goals get into the product. We still have a long way to go with many years of exciting development ahead of us, but the foundation is pretty much rock solid for what has become the “Delphi” of our age.

Working hard on the update

Binary at last

Those of you that follow my blog know that real data, binary data, is a touchy subject for JavaScript programmers. This has motivated me to work even harder to bypass and overcome the challenges the platform represents. And needless to say, we have not just overcome the barriers, but we have managed to do so without turning to hacks or parlor tricks involving strings.

We are in fact the only HTML5 compiler suite to offer a 1:1 stream system 100% compatible with native languages. In fact, we even modeled the stream classes themselves after FreePascal and Delphi. This means that not only do they work as you expect, they also are exactly what you expect. Right down to the last method.

This has changed Smart Mobile Studio for my part. I loved working with SMS since the day it was born, but with the advent of memory allocations, marshaled pointers, streams and memory IO which gives native object pascal a run for it’s money, Smart Mobile Studio enters a new era.

More memory tools

The memory system in FreePascal and Delphi is rich. You have out of the box support for Allocmem and all the low-level methods that have been there since the last days of Turbo Pascal, followed by streams of various types, reader and writer isolation and a fast binary persistent framework: TFiler.

Well, we decided to do some changes to this old recipe. It’s not every day you get to write an object pascal RTL from scratch and define a language and dialect, so trying to get this right was tricky. You want to be backwards compatible, but not so much as to hinder future development and changes.

As of writing the code has been divided into the following sections

• Unmanaged memory
• Managed memory
• Streams
• Binary data
• Datatype conversion

Unmanaged memory

As the name implies, this represents classes and methods for allocating memory which are un-managed by Smart Pascal classes. JavaScript has built-in garbage collection so we are not talking about that type of management, but rather management which comes into play when you use the same buffer from multiple places. Should the segment be cloned? Should it be split? This is what management is all about. TUnManaged and TMarshal are objects covering these topics.

Managed memory

This includes marshaled pointers, which takes care of references to memory segments and offset’s into those segments. This emulates the mechanics of a typed-pointer under FreePascal or Delphi. Associated with these are classes and methods allowing you to allocate, release, scale and move data between segments.

Streams

As of writing Smart Mobile Studio supports TStream, TMemoryStream, TStringStream, TReader and TWriters. More streams will be added, including ZLib compression and probably an extension to attach encryption codec’s.

Binary Data

The ability to convert intrinsic datatypes into bytes and bytes back to intrinsic datatypes. Also directly to un-typed memory segments. This is covered by TDatatype and various other classes. Also the core class, TBinaryData, which inherits from TAllocation provides a plethora of methods for altering memory segments, all the way down to bit operations.

Dataset and databases

Dataset + NodeJS = true

This is the big one that many people are waiting for. Well, the update will not provide wizards and 1-2-3 connection steps for popular databases. That requires a bit more infrastructure (but we are working on such wizards, so dont worry, it’s on its way). But this is the point in time where we introduce TW3Dataset, which is an in-memory database table.

I have just added the methods required to store directly to binary format (streams). It supports JSON format already, so now we have a solution for both JS friendly code and native systems.

The thing about the dataset is that it’s tailored after Mono/C#’s dataset. Meaning that it’s ultra-light, fast, simple to use and easy to deploy. Perfect for sending data between clients and servers, not to mention in-memory caching of information. If you are coding a web-shop you would use a TW3Dataset in memory to keep track of products ordered, and then transport the entire dataset to your nodeJS or RemObjects SDK service when the customer clicks “Order”.

TW3Dataset is also written in Smart Mobile Studio. It’s not a wrapper around WebSQL or IndexDB which are severely limited. Want to persist it in the browser? No problem. Just save the dataset into LocalStorage and re-load it the next time a customer visits your website.

So now you can do cool stuff like:

uses System.Types, System.TypeConv, System.Memory, System.Streams, System.Dataset;

Procedure TApplication.InitializeObject;
begin
  inherited;
  //Create our application global dataset
  FAppTable:=TW3Dataset.Create;

  //Do we have data in browser storage? If so, load
  if FStorage.keyExists("appdata") then
  FAppTable.LoadFromStream(Storage.KeyToStream("AppData")) else
  begin
    //First visit or storage deleted, re-create table
    FApptable.FieldDefs.Add("id",ftAutoInc);
    FAppTable.FieldDefs.Add("session_key",ftGUID);
    FAppTable.fieldDefs.Add("customer_id",ftInteger);
    FAppTable.fieldDefs.add("userid",ftString);
    FAppTable.FieldDefs.add("last_login",ftDateTime);
    FAppTable.FieldDefs.add("shopping_basket",ftBlob);
    FAppTable.CreateDataset;
  end;
end;

I took the liberty of adding auto-generated fields, such as ftAutoInc and ftGUID. A GUID field is excellent for HTML5 development, because dictionaries in the world of JavaScript are name/value pairs. So GUIDs ensure entries have unique identifiers for serialization.

Another cool factor which is now possible to achieve, is ZLIB compression and power encryption (like RC4, Blowfish etc). As you know there is a limit to how much data you are allowed to store via the browser. The capacity varies from browser to browser but is typically in the 5-10 megabyte range. Thankfully our TW3Dataset is as lightweight as possible and will be able to pack a lot of data into 10 megabytes is structured properly.

It’s important to underline that this limitation only exists when you embed Smart Mobile applications into your webpage (like you would embed Flash applications earlier). The moment you compile your Smart Pascal application with Cordova / Phonegap, the limitation goes away and you have the same storage rights as other native applications.

But using Smart Mobile Studio as a complete replacement for Adobe Flash, or even better – write your entire website in Smart Mobile Studio in order to de-couple the presentation from the data layer, is now easier than ever before. Why spend hundreds of dollars on buying sliding banners and other website bling, when you can code it yourself in a couple of hours in Smart?

Kick ass compression

ZLIB compression is going to help with many aspects of web application development. For instance it will boost your ability for record caching; sending records in bulk back to the company server as opposed to calling the server once every time a customer adds a product to his basket.

In our model, the customer selects their products and these are stored in a TW3Dataset. Only when the customer is ready to use his or her’s credit-card and click the “Pay now” button should you transport the full dataset to the company servers for processing. If you are worried about safety, this is actually more safe than the majority of existing solutions. You would naturally run the page with SSL, disable source-code access etc. And by applying encryption and compression to your data, it’s going to be very hard for people to make use if it.

But what should you run server-side? Well, this is where Smart Mobile Studio’s support for NodeJS comes in. Because you write the server as well in Smart Pascal. NodeJS has excellent support for all types of databases (Oracle, MSSQL, MySQL, MariaDB and a bunch of others) so connecting to your database is easy enough.

All you have to write is the REST API you want to expose to your mobile or HTML5 web-application. With that in place you have effectively solves the entire IT infrastructure from A to Z with a single technology: Object Pascal.

Well — back to the code!

JS script engine for your Smart programs?

Native object pascal has a plethora of scripting engines attached to it. You have DWScript, Pascal Script and a whole host of homebrew languages which you can embed to your FreePascal or Delphi applications.

Well now you can do the same for Smart Mobile Studio projects!

The Acorn JavaScript parser is written in JavaScript itself, and has been expanded with a runtime module. Which means you can now execute JavaScript from JavaScript. If this sounds odd, consider the fact that JavaScript has no threading, so this is probably the closest you get to “real” threads.

Script engines ohoy! Now for HTML5 and Smart Pascal

The acorn engine allows you to load, parse and then execute a script either all at once, or by executing a small fraction at a time. Which is perfect for a timer type execution where you can run several scripts at once through timers. This is how systems like Windows work with real processes (where the CPU quickly jumps between processes and executes a “slice” of machine code on each).

What do I need?

Simple. First, download the file “acorn_interpreter.js” from github. Save that file into the $SMART\Libraries folder (see the shortcuts on your start-menu).

Here is the code which wraps the engine. Enjoy!

unit acornJS;

interface

uses
  SmartCL.System;

type

  TAcornInterpreter = Class(TObject)
  private
    FHandle:    THandle;
    FSource:    String;
  protected
    procedure   setSource(Value:String);
  public
    Property    Active:Boolean read ( FHandle<>Null );
    Property    Source:String read FSource write setSource;
    procedure   LoadFromString(aSource:String);
    function    Step:Boolean;
    procedure   Prepare;
    Destructor  Destroy;Override;
  end;

implementation

{$r "acorn_interpreter.js"}

//############################################################################
// TAcornInterpreter
//############################################################################

Destructor TAcornInterpreter.Destroy;
begin
  if (FHandle) then
  FHandle := NIL;
  inherited;
end;

procedure TAcornInterpreter.setSource(Value:String);
begin
  if (FHandle) then
  FHandle:=NULL;
  FSource:=Value;
end;

procedure TAcornInterpreter.LoadFromString(aSource:String);
begin
  setSource(aSource);
end;

function TAcornInterpreter.Step:Boolean;
begin
  if (FHandle) then
  result:=FHandle.step();
end;

Procedure TAcornInterpreter.Prepare;
var
  mRef: THandle;
begin
  asm
    @mRef = new Interpreter(@source);
  end;
  FHandle:=mRef;
  mRef:=null;
end;
end.

It’s just a trick, it must be, or?

Some of you may think “oh it’s just another EVAL routine which doesn’t really parse JavaScript from scratch”. Well I am happy to inform you that you are mistaken. It is a full parser which generates an AST – as can be seen below.

Note: This version is a bit dated, so get a fresh copy of the latest acorn engine from the git repo’s listed in the sources below!

// Acorn is a tiny, fast JavaScript parser written in JavaScript.
//
// Acorn was written by Marijn Haverbeke and released under an MIT
// license. The Unicode regexps (for identifiers and whitespace) were
// taken from [Esprima](http://esprima.org) by Ariya Hidayat.
//
// Git repositories for Acorn are available at
//
//     http://marijnhaverbeke.nl/git/acorn
//     https://github.com/marijnh/acorn.git
//
// Please use the [github bug tracker][ghbt] to report issues.
//
// [ghbt]: https://github.com/marijnh/acorn/issues
//
// This file defines the main parser interface. The library also comes
// with a [error-tolerant parser][dammit] and an
// [abstract syntax tree walker][walk], defined in other files.
//
// [dammit]: acorn_loose.js
// [walk]: util/walk.js

(function(root, mod) {
  if (typeof exports == "object" && typeof module == "object") return mod(exports); // CommonJS
  if (typeof define == "function" && define.amd) return define(["exports"], mod); // AMD
  mod(root.acorn || (root.acorn = {})); // Plain browser env
})(this, function(exports) {
  "use strict";

  exports.version = "0.4.1";

  // The main exported interface (under `self.acorn` when in the
  // browser) is a `parse` function that takes a code string and
  // returns an abstract syntax tree as specified by [Mozilla parser
  // API][api], with the caveat that the SpiderMonkey-specific syntax
  // (`let`, `yield`, inline XML, etc) is not recognized.
  //
  // [api]: https://developer.mozilla.org/en-US/docs/SpiderMonkey/Parser_API

  var options, input, inputLen, sourceFile;

  exports.parse = function(inpt, opts) {
    input = String(inpt); inputLen = input.length;
    setOptions(opts);
    initTokenState();
    return parseTopLevel(options.program);
  };

  // A second optional argument can be given to further configure
  // the parser process. These options are recognized:

  var defaultOptions = exports.defaultOptions = {
    // `ecmaVersion` indicates the ECMAScript version to parse. Must
    // be either 3 or 5. This
    // influences support for strict mode, the set of reserved words, and
    // support for getters and setter.
    ecmaVersion: 5,
    // Turn on `strictSemicolons` to prevent the parser from doing
    // automatic semicolon insertion.
    strictSemicolons: false,
    // When `allowTrailingCommas` is false, the parser will not allow
    // trailing commas in array and object literals.
    allowTrailingCommas: true,
    // By default, reserved words are not enforced. Enable
    // `forbidReserved` to enforce them.
    forbidReserved: false,
    // When `locations` is on, `loc` properties holding objects with
    // `start` and `end` properties in `{line, column}` form (with
    // line being 1-based and column 0-based) will be attached to the
    // nodes.
    locations: false,
    // A function can be passed as `onComment` option, which will
    // cause Acorn to call that function with `(block, text, start,
    // end)` parameters whenever a comment is skipped. `block` is a
    // boolean indicating whether this is a block (`/* */`) comment,
    // `text` is the content of the comment, and `start` and `end` are
    // character offsets that denote the start and end of the comment.
    // When the `locations` option is on, two more parameters are
    // passed, the full `{line, column}` locations of the start and
    // end of the comments.
    onComment: null,
    // Nodes have their start and end characters offsets recorded in
    // `start` and `end` properties (directly on the node, rather than
    // the `loc` object, which holds line/column data. To also add a
    // [semi-standardized][range] `range` property holding a `[start,
    // end]` array with the same numbers, set the `ranges` option to
    // `true`.
    //
    // [range]: https://bugzilla.mozilla.org/show_bug.cgi?id=745678
    ranges: false,
    // It is possible to parse multiple files into a single AST by
    // passing the tree produced by parsing the first file as
    // `program` option in subsequent parses. This will add the
    // toplevel forms of the parsed file to the `Program` (top) node
    // of an existing parse tree.
    program: null,
    // When `location` is on, you can pass this to record the source
    // file in every node's `loc` object.
    sourceFile: null,
    // This value, if given, is stored in every node, whether
    // `location` is on or off.
    directSourceFile: null
  };

  function setOptions(opts) {
    options = opts || {};
    for (var opt in defaultOptions) if (!Object.prototype.hasOwnProperty.call(options, opt))
      options[opt] = defaultOptions[opt];
    sourceFile = options.sourceFile || null;
  }

  // The `getLineInfo` function is mostly useful when the
  // `locations` option is off (for performance reasons) and you
  // want to find the line/column position for a given character
  // offset. `input` should be the code string that the offset refers
  // into.

  var getLineInfo = exports.getLineInfo = function(input, offset) {
    for (var line = 1, cur = 0;;) {
      lineBreak.lastIndex = cur;
      var match = lineBreak.exec(input);
      if (match && match.index < offset) {
        ++line;
        cur = match.index + match[0].length;
      } else break;
    }
    return {line: line, column: offset - cur};
  };

  // Acorn is organized as a tokenizer and a recursive-descent parser.
  // The `tokenize` export provides an interface to the tokenizer.
  // Because the tokenizer is optimized for being efficiently used by
  // the Acorn parser itself, this interface is somewhat crude and not
  // very modular. Performing another parse or call to `tokenize` will
  // reset the internal state, and invalidate existing tokenizers.

  exports.tokenize = function(inpt, opts) {
    input = String(inpt); inputLen = input.length;
    setOptions(opts);
    initTokenState();

    var t = {};
    function getToken(forceRegexp) {
      readToken(forceRegexp);
      t.start = tokStart; t.end = tokEnd;
      t.startLoc = tokStartLoc; t.endLoc = tokEndLoc;
      t.type = tokType; t.value = tokVal;
      return t;
    }
    getToken.jumpTo = function(pos, reAllowed) {
      tokPos = pos;
      if (options.locations) {
        tokCurLine = 1;
        tokLineStart = lineBreak.lastIndex = 0;
        var match;
        while ((match = lineBreak.exec(input)) && match.index < pos) {
          ++tokCurLine;
          tokLineStart = match.index + match[0].length;
        }
      }
      tokRegexpAllowed = reAllowed;
      skipSpace();
    };
    return getToken;
  };

  // State is kept in (closure-)global variables. We already saw the
  // `options`, `input`, and `inputLen` variables above.

  // The current position of the tokenizer in the input.

  var tokPos;

  // The start and end offsets of the current token.

  var tokStart, tokEnd;

  // When `options.locations` is true, these hold objects
  // containing the tokens start and end line/column pairs.

  var tokStartLoc, tokEndLoc;

  // The type and value of the current token. Token types are objects,
  // named by variables against which they can be compared, and
  // holding properties that describe them (indicating, for example,
  // the precedence of an infix operator, and the original name of a
  // keyword token). The kind of value that's held in `tokVal` depends
  // on the type of the token. For literals, it is the literal value,
  // for operators, the operator name, and so on.

  var tokType, tokVal;

  // Interal state for the tokenizer. To distinguish between division
  // operators and regular expressions, it remembers whether the last
  // token was one that is allowed to be followed by an expression.
  // (If it is, a slash is probably a regexp, if it isn't it's a
  // division operator. See the `parseStatement` function for a
  // caveat.)

  var tokRegexpAllowed;

  // When `options.locations` is true, these are used to keep
  // track of the current line, and know when a new line has been
  // entered.

  var tokCurLine, tokLineStart;

  // These store the position of the previous token, which is useful
  // when finishing a node and assigning its `end` position.

  var lastStart, lastEnd, lastEndLoc;

  // This is the parser's state. `inFunction` is used to reject
  // `return` statements outside of functions, `labels` to verify that
  // `break` and `continue` have somewhere to jump to, and `strict`
  // indicates whether strict mode is on.

  var inFunction, labels, strict;

  // This function is used to raise exceptions on parse errors. It
  // takes an offset integer (into the current `input`) to indicate
  // the location of the error, attaches the position to the end
  // of the error message, and then raises a `SyntaxError` with that
  // message.

  function raise(pos, message) {
    var loc = getLineInfo(input, pos);
    message += " (" + loc.line + ":" + loc.column + ")";
    var err = new SyntaxError(message);
    err.pos = pos; err.loc = loc; err.raisedAt = tokPos;
    throw err;
  }

  // Reused empty array added for node fields that are always empty.

  var empty = [];

  // ## Token types

  // The assignment of fine-grained, information-carrying type objects
  // allows the tokenizer to store the information it has about a
  // token in a way that is very cheap for the parser to look up.

  // All token type variables start with an underscore, to make them
  // easy to recognize.

  // These are the general types. The `type` property is only used to
  // make them recognizeable when debugging.

  var _num = {type: "num"}, _regexp = {type: "regexp"}, _string = {type: "string"};
  var _name = {type: "name"}, _eof = {type: "eof"};

  // Keyword tokens. The `keyword` property (also used in keyword-like
  // operators) indicates that the token originated from an
  // identifier-like word, which is used when parsing property names.
  //
  // The `beforeExpr` property is used to disambiguate between regular
  // expressions and divisions. It is set on all token types that can
  // be followed by an expression (thus, a slash after them would be a
  // regular expression).
  //
  // `isLoop` marks a keyword as starting a loop, which is important
  // to know when parsing a label, in order to allow or disallow
  // continue jumps to that label.

  var _break = {keyword: "break"}, _case = {keyword: "case", beforeExpr: true}, _catch = {keyword: "catch"};
  var _continue = {keyword: "continue"}, _debugger = {keyword: "debugger"}, _default = {keyword: "default"};
  var _do = {keyword: "do", isLoop: true}, _else = {keyword: "else", beforeExpr: true};
  var _finally = {keyword: "finally"}, _for = {keyword: "for", isLoop: true}, _function = {keyword: "function"};
  var _if = {keyword: "if"}, _return = {keyword: "return", beforeExpr: true}, _switch = {keyword: "switch"};
  var _throw = {keyword: "throw", beforeExpr: true}, _try = {keyword: "try"}, _var = {keyword: "var"};
  var _while = {keyword: "while", isLoop: true}, _with = {keyword: "with"}, _new = {keyword: "new", beforeExpr: true};
  var _this = {keyword: "this"};

  // The keywords that denote values.

  var _null = {keyword: "null", atomValue: null}, _true = {keyword: "true", atomValue: true};
  var _false = {keyword: "false", atomValue: false};

  // Some keywords are treated as regular operators. `in` sometimes
  // (when parsing `for`) needs to be tested against specifically, so
  // we assign a variable name to it for quick comparing.

  var _in = {keyword: "in", binop: 7, beforeExpr: true};

  // Map keyword names to token types.

  var keywordTypes = {"break": _break, "case": _case, "catch": _catch,
                      "continue": _continue, "debugger": _debugger, "default": _default,
                      "do": _do, "else": _else, "finally": _finally, "for": _for,
                      "function": _function, "if": _if, "return": _return, "switch": _switch,
                      "throw": _throw, "try": _try, "var": _var, "while": _while, "with": _with,
                      "null": _null, "true": _true, "false": _false, "new": _new, "in": _in,
                      "instanceof": {keyword: "instanceof", binop: 7, beforeExpr: true}, "this": _this,
                      "typeof": {keyword: "typeof", prefix: true, beforeExpr: true},
                      "void": {keyword: "void", prefix: true, beforeExpr: true},
                      "delete": {keyword: "delete", prefix: true, beforeExpr: true}};

  // Punctuation token types. Again, the `type` property is purely for debugging.

  var _bracketL = {type: "[", beforeExpr: true}, _bracketR = {type: "]"}, _braceL = {type: "{", beforeExpr: true};
  var _braceR = {type: "}"}, _parenL = {type: "(", beforeExpr: true}, _parenR = {type: ")"};
  var _comma = {type: ",", beforeExpr: true}, _semi = {type: ";", beforeExpr: true};
  var _colon = {type: ":", beforeExpr: true}, _dot = {type: "."}, _question = {type: "?", beforeExpr: true};

  // Operators. These carry several kinds of properties to help the
  // parser use them properly (the presence of these properties is
  // what categorizes them as operators).
  //
  // `binop`, when present, specifies that this operator is a binary
  // operator, and will refer to its precedence.
  //
  // `prefix` and `postfix` mark the operator as a prefix or postfix
  // unary operator. `isUpdate` specifies that the node produced by
  // the operator should be of type UpdateExpression rather than
  // simply UnaryExpression (`++` and `--`).
  //
  // `isAssign` marks all of `=`, `+=`, `-=` etcetera, which act as
  // binary operators with a very low precedence, that should result
  // in AssignmentExpression nodes.

  var _slash = {binop: 10, beforeExpr: true}, _eq = {isAssign: true, beforeExpr: true};
  var _assign = {isAssign: true, beforeExpr: true};
  var _incDec = {postfix: true, prefix: true, isUpdate: true}, _prefix = {prefix: true, beforeExpr: true};
  var _logicalOR = {binop: 1, beforeExpr: true};
  var _logicalAND = {binop: 2, beforeExpr: true};
  var _bitwiseOR = {binop: 3, beforeExpr: true};
  var _bitwiseXOR = {binop: 4, beforeExpr: true};
  var _bitwiseAND = {binop: 5, beforeExpr: true};
  var _equality = {binop: 6, beforeExpr: true};
  var _relational = {binop: 7, beforeExpr: true};
  var _bitShift = {binop: 8, beforeExpr: true};
  var _plusMin = {binop: 9, prefix: true, beforeExpr: true};
  var _multiplyModulo = {binop: 10, beforeExpr: true};

  // Provide access to the token types for external users of the
  // tokenizer.

  exports.tokTypes = {bracketL: _bracketL, bracketR: _bracketR, braceL: _braceL, braceR: _braceR,
                      parenL: _parenL, parenR: _parenR, comma: _comma, semi: _semi, colon: _colon,
                      dot: _dot, question: _question, slash: _slash, eq: _eq, name: _name, eof: _eof,
                      num: _num, regexp: _regexp, string: _string};
  for (var kw in keywordTypes) exports.tokTypes["_" + kw] = keywordTypes[kw];

  // This is a trick taken from Esprima. It turns out that, on
  // non-Chrome browsers, to check whether a string is in a set, a
  // predicate containing a big ugly `switch` statement is faster than
  // a regular expression, and on Chrome the two are about on par.
  // This function uses `eval` (non-lexical) to produce such a
  // predicate from a space-separated string of words.
  //
  // It starts by sorting the words by length.

  function makePredicate(words) {
    words = words.split(" ");
    var f = "", cats = [];
    out: for (var i = 0; i < words.length; ++i) {
      for (var j = 0; j < cats.length; ++j)
        if (cats[j][0].length == words[i].length) {
          cats[j].push(words[i]);
          continue out;
        }
      cats.push([words[i]]);
    }
    function compareTo(arr) {
      if (arr.length == 1) return f += "return str === " + JSON.stringify(arr[0]) + ";";
      f += "switch(str){";
      for (var i = 0; i < arr.length; ++i) f += "case " + JSON.stringify(arr[i]) + ":";
      f += "return true}return false;";
    }

    // When there are more than three length categories, an outer
    // switch first dispatches on the lengths, to save on comparisons.

    if (cats.length > 3) {
      cats.sort(function(a, b) {return b.length - a.length;});
      f += "switch(str.length){";
      for (var i = 0; i < cats.length; ++i) {
        var cat = cats[i];
        f += "case " + cat[0].length + ":";
        compareTo(cat);
      }
      f += "}";

    // Otherwise, simply generate a flat `switch` statement.

    } else {
      compareTo(words);
    }
    return new Function("str", f);
  }

  // The ECMAScript 3 reserved word list.

  var isReservedWord3 = makePredicate("abstract boolean byte char class double enum export extends final float goto implements import int interface long native package private protected public short static super synchronized throws transient volatile");

  // ECMAScript 5 reserved words.

  var isReservedWord5 = makePredicate("class enum extends super const export import");

  // The additional reserved words in strict mode.

  var isStrictReservedWord = makePredicate("implements interface let package private protected public static yield");

  // The forbidden variable names in strict mode.

  var isStrictBadIdWord = makePredicate("eval arguments");

  // And the keywords.

  var isKeyword = makePredicate("break case catch continue debugger default do else finally for function if return switch throw try var while with null true false instanceof typeof void delete new in this");

  // ## Character categories

  // Big ugly regular expressions that match characters in the
  // whitespace, identifier, and identifier-start categories. These
  // are only applied when a character is found to actually have a
  // code point above 128.

  var nonASCIIwhitespace = /[\u1680\u180e\u2000-\u200a\u202f\u205f\u3000\ufeff]/;
  var nonASCIIidentifierStartChars = "\xaa\xb5\xba\xc0-\xd6\xd8-\xf6\xf8-\u02c1\u02c6-\u02d1\u02e0-\u02e4\u02ec\u02ee\u0370-\u0374\u0376\u0377\u037a-\u037d\u0386\u0388-\u038a\u038c\u038e-\u03a1\u03a3-\u03f5\u03f7-\u0481\u048a-\u0527\u0531-\u0556\u0559\u0561-\u0587\u05d0-\u05ea\u05f0-\u05f2\u0620-\u064a\u066e\u066f\u0671-\u06d3\u06d5\u06e5\u06e6\u06ee\u06ef\u06fa-\u06fc\u06ff\u0710\u0712-\u072f\u074d-\u07a5\u07b1\u07ca-\u07ea\u07f4\u07f5\u07fa\u0800-\u0815\u081a\u0824\u0828\u0840-\u0858\u08a0\u08a2-\u08ac\u0904-\u0939\u093d\u0950\u0958-\u0961\u0971-\u0977\u0979-\u097f\u0985-\u098c\u098f\u0990\u0993-\u09a8\u09aa-\u09b0\u09b2\u09b6-\u09b9\u09bd\u09ce\u09dc\u09dd\u09df-\u09e1\u09f0\u09f1\u0a05-\u0a0a\u0a0f\u0a10\u0a13-\u0a28\u0a2a-\u0a30\u0a32\u0a33\u0a35\u0a36\u0a38\u0a39\u0a59-\u0a5c\u0a5e\u0a72-\u0a74\u0a85-\u0a8d\u0a8f-\u0a91\u0a93-\u0aa8\u0aaa-\u0ab0\u0ab2\u0ab3\u0ab5-\u0ab9\u0abd\u0ad0\u0ae0\u0ae1\u0b05-\u0b0c\u0b0f\u0b10\u0b13-\u0b28\u0b2a-\u0b30\u0b32\u0b33\u0b35-\u0b39\u0b3d\u0b5c\u0b5d\u0b5f-\u0b61\u0b71\u0b83\u0b85-\u0b8a\u0b8e-\u0b90\u0b92-\u0b95\u0b99\u0b9a\u0b9c\u0b9e\u0b9f\u0ba3\u0ba4\u0ba8-\u0baa\u0bae-\u0bb9\u0bd0\u0c05-\u0c0c\u0c0e-\u0c10\u0c12-\u0c28\u0c2a-\u0c33\u0c35-\u0c39\u0c3d\u0c58\u0c59\u0c60\u0c61\u0c85-\u0c8c\u0c8e-\u0c90\u0c92-\u0ca8\u0caa-\u0cb3\u0cb5-\u0cb9\u0cbd\u0cde\u0ce0\u0ce1\u0cf1\u0cf2\u0d05-\u0d0c\u0d0e-\u0d10\u0d12-\u0d3a\u0d3d\u0d4e\u0d60\u0d61\u0d7a-\u0d7f\u0d85-\u0d96\u0d9a-\u0db1\u0db3-\u0dbb\u0dbd\u0dc0-\u0dc6\u0e01-\u0e30\u0e32\u0e33\u0e40-\u0e46\u0e81\u0e82\u0e84\u0e87\u0e88\u0e8a\u0e8d\u0e94-\u0e97\u0e99-\u0e9f\u0ea1-\u0ea3\u0ea5\u0ea7\u0eaa\u0eab\u0ead-\u0eb0\u0eb2\u0eb3\u0ebd\u0ec0-\u0ec4\u0ec6\u0edc-\u0edf\u0f00\u0f40-\u0f47\u0f49-\u0f6c\u0f88-\u0f8c\u1000-\u102a\u103f\u1050-\u1055\u105a-\u105d\u1061\u1065\u1066\u106e-\u1070\u1075-\u1081\u108e\u10a0-\u10c5\u10c7\u10cd\u10d0-\u10fa\u10fc-\u1248\u124a-\u124d\u1250-\u1256\u1258\u125a-\u125d\u1260-\u1288\u128a-\u128d\u1290-\u12b0\u12b2-\u12b5\u12b8-\u12be\u12c0\u12c2-\u12c5\u12c8-\u12d6\u12d8-\u1310\u1312-\u1315\u1318-\u135a\u1380-\u138f\u13a0-\u13f4\u1401-\u166c\u166f-\u167f\u1681-\u169a\u16a0-\u16ea\u16ee-\u16f0\u1700-\u170c\u170e-\u1711\u1720-\u1731\u1740-\u1751\u1760-\u176c\u176e-\u1770\u1780-\u17b3\u17d7\u17dc\u1820-\u1877\u1880-\u18a8\u18aa\u18b0-\u18f5\u1900-\u191c\u1950-\u196d\u1970-\u1974\u1980-\u19ab\u19c1-\u19c7\u1a00-\u1a16\u1a20-\u1a54\u1aa7\u1b05-\u1b33\u1b45-\u1b4b\u1b83-\u1ba0\u1bae\u1baf\u1bba-\u1be5\u1c00-\u1c23\u1c4d-\u1c4f\u1c5a-\u1c7d\u1ce9-\u1cec\u1cee-\u1cf1\u1cf5\u1cf6\u1d00-\u1dbf\u1e00-\u1f15\u1f18-\u1f1d\u1f20-\u1f45\u1f48-\u1f4d\u1f50-\u1f57\u1f59\u1f5b\u1f5d\u1f5f-\u1f7d\u1f80-\u1fb4\u1fb6-\u1fbc\u1fbe\u1fc2-\u1fc4\u1fc6-\u1fcc\u1fd0-\u1fd3\u1fd6-\u1fdb\u1fe0-\u1fec\u1ff2-\u1ff4\u1ff6-\u1ffc\u2071\u207f\u2090-\u209c\u2102\u2107\u210a-\u2113\u2115\u2119-\u211d\u2124\u2126\u2128\u212a-\u212d\u212f-\u2139\u213c-\u213f\u2145-\u2149\u214e\u2160-\u2188\u2c00-\u2c2e\u2c30-\u2c5e\u2c60-\u2ce4\u2ceb-\u2cee\u2cf2\u2cf3\u2d00-\u2d25\u2d27\u2d2d\u2d30-\u2d67\u2d6f\u2d80-\u2d96\u2da0-\u2da6\u2da8-\u2dae\u2db0-\u2db6\u2db8-\u2dbe\u2dc0-\u2dc6\u2dc8-\u2dce\u2dd0-\u2dd6\u2dd8-\u2dde\u2e2f\u3005-\u3007\u3021-\u3029\u3031-\u3035\u3038-\u303c\u3041-\u3096\u309d-\u309f\u30a1-\u30fa\u30fc-\u30ff\u3105-\u312d\u3131-\u318e\u31a0-\u31ba\u31f0-\u31ff\u3400-\u4db5\u4e00-\u9fcc\ua000-\ua48c\ua4d0-\ua4fd\ua500-\ua60c\ua610-\ua61f\ua62a\ua62b\ua640-\ua66e\ua67f-\ua697\ua6a0-\ua6ef\ua717-\ua71f\ua722-\ua788\ua78b-\ua78e\ua790-\ua793\ua7a0-\ua7aa\ua7f8-\ua801\ua803-\ua805\ua807-\ua80a\ua80c-\ua822\ua840-\ua873\ua882-\ua8b3\ua8f2-\ua8f7\ua8fb\ua90a-\ua925\ua930-\ua946\ua960-\ua97c\ua984-\ua9b2\ua9cf\uaa00-\uaa28\uaa40-\uaa42\uaa44-\uaa4b\uaa60-\uaa76\uaa7a\uaa80-\uaaaf\uaab1\uaab5\uaab6\uaab9-\uaabd\uaac0\uaac2\uaadb-\uaadd\uaae0-\uaaea\uaaf2-\uaaf4\uab01-\uab06\uab09-\uab0e\uab11-\uab16\uab20-\uab26\uab28-\uab2e\uabc0-\uabe2\uac00-\ud7a3\ud7b0-\ud7c6\ud7cb-\ud7fb\uf900-\ufa6d\ufa70-\ufad9\ufb00-\ufb06\ufb13-\ufb17\ufb1d\ufb1f-\ufb28\ufb2a-\ufb36\ufb38-\ufb3c\ufb3e\ufb40\ufb41\ufb43\ufb44\ufb46-\ufbb1\ufbd3-\ufd3d\ufd50-\ufd8f\ufd92-\ufdc7\ufdf0-\ufdfb\ufe70-\ufe74\ufe76-\ufefc\uff21-\uff3a\uff41-\uff5a\uff66-\uffbe\uffc2-\uffc7\uffca-\uffcf\uffd2-\uffd7\uffda-\uffdc";
  var nonASCIIidentifierChars = "\u0300-\u036f\u0483-\u0487\u0591-\u05bd\u05bf\u05c1\u05c2\u05c4\u05c5\u05c7\u0610-\u061a\u0620-\u0649\u0672-\u06d3\u06e7-\u06e8\u06fb-\u06fc\u0730-\u074a\u0800-\u0814\u081b-\u0823\u0825-\u0827\u0829-\u082d\u0840-\u0857\u08e4-\u08fe\u0900-\u0903\u093a-\u093c\u093e-\u094f\u0951-\u0957\u0962-\u0963\u0966-\u096f\u0981-\u0983\u09bc\u09be-\u09c4\u09c7\u09c8\u09d7\u09df-\u09e0\u0a01-\u0a03\u0a3c\u0a3e-\u0a42\u0a47\u0a48\u0a4b-\u0a4d\u0a51\u0a66-\u0a71\u0a75\u0a81-\u0a83\u0abc\u0abe-\u0ac5\u0ac7-\u0ac9\u0acb-\u0acd\u0ae2-\u0ae3\u0ae6-\u0aef\u0b01-\u0b03\u0b3c\u0b3e-\u0b44\u0b47\u0b48\u0b4b-\u0b4d\u0b56\u0b57\u0b5f-\u0b60\u0b66-\u0b6f\u0b82\u0bbe-\u0bc2\u0bc6-\u0bc8\u0bca-\u0bcd\u0bd7\u0be6-\u0bef\u0c01-\u0c03\u0c46-\u0c48\u0c4a-\u0c4d\u0c55\u0c56\u0c62-\u0c63\u0c66-\u0c6f\u0c82\u0c83\u0cbc\u0cbe-\u0cc4\u0cc6-\u0cc8\u0cca-\u0ccd\u0cd5\u0cd6\u0ce2-\u0ce3\u0ce6-\u0cef\u0d02\u0d03\u0d46-\u0d48\u0d57\u0d62-\u0d63\u0d66-\u0d6f\u0d82\u0d83\u0dca\u0dcf-\u0dd4\u0dd6\u0dd8-\u0ddf\u0df2\u0df3\u0e34-\u0e3a\u0e40-\u0e45\u0e50-\u0e59\u0eb4-\u0eb9\u0ec8-\u0ecd\u0ed0-\u0ed9\u0f18\u0f19\u0f20-\u0f29\u0f35\u0f37\u0f39\u0f41-\u0f47\u0f71-\u0f84\u0f86-\u0f87\u0f8d-\u0f97\u0f99-\u0fbc\u0fc6\u1000-\u1029\u1040-\u1049\u1067-\u106d\u1071-\u1074\u1082-\u108d\u108f-\u109d\u135d-\u135f\u170e-\u1710\u1720-\u1730\u1740-\u1750\u1772\u1773\u1780-\u17b2\u17dd\u17e0-\u17e9\u180b-\u180d\u1810-\u1819\u1920-\u192b\u1930-\u193b\u1951-\u196d\u19b0-\u19c0\u19c8-\u19c9\u19d0-\u19d9\u1a00-\u1a15\u1a20-\u1a53\u1a60-\u1a7c\u1a7f-\u1a89\u1a90-\u1a99\u1b46-\u1b4b\u1b50-\u1b59\u1b6b-\u1b73\u1bb0-\u1bb9\u1be6-\u1bf3\u1c00-\u1c22\u1c40-\u1c49\u1c5b-\u1c7d\u1cd0-\u1cd2\u1d00-\u1dbe\u1e01-\u1f15\u200c\u200d\u203f\u2040\u2054\u20d0-\u20dc\u20e1\u20e5-\u20f0\u2d81-\u2d96\u2de0-\u2dff\u3021-\u3028\u3099\u309a\ua640-\ua66d\ua674-\ua67d\ua69f\ua6f0-\ua6f1\ua7f8-\ua800\ua806\ua80b\ua823-\ua827\ua880-\ua881\ua8b4-\ua8c4\ua8d0-\ua8d9\ua8f3-\ua8f7\ua900-\ua909\ua926-\ua92d\ua930-\ua945\ua980-\ua983\ua9b3-\ua9c0\uaa00-\uaa27\uaa40-\uaa41\uaa4c-\uaa4d\uaa50-\uaa59\uaa7b\uaae0-\uaae9\uaaf2-\uaaf3\uabc0-\uabe1\uabec\uabed\uabf0-\uabf9\ufb20-\ufb28\ufe00-\ufe0f\ufe20-\ufe26\ufe33\ufe34\ufe4d-\ufe4f\uff10-\uff19\uff3f";
  var nonASCIIidentifierStart = new RegExp("[" + nonASCIIidentifierStartChars + "]");
  var nonASCIIidentifier = new RegExp("[" + nonASCIIidentifierStartChars + nonASCIIidentifierChars + "]");

  // Whether a single character denotes a newline.

  var newline = /[\n\r\u2028\u2029]/;

  // Matches a whole line break (where CRLF is considered a single
  // line break). Used to count lines.

  var lineBreak = /\r\n|[\n\r\u2028\u2029]/g;

  // Test whether a given character code starts an identifier.

  var isIdentifierStart = exports.isIdentifierStart = function(code) {
    if (code < 65) return code === 36;
    if (code < 91) return true;
    if (code < 97) return code === 95;
    if (code < 123)return true;
    return code >= 0xaa && nonASCIIidentifierStart.test(String.fromCharCode(code));
  };

  // Test whether a given character is part of an identifier.

  var isIdentifierChar = exports.isIdentifierChar = function(code) {
    if (code < 48) return code === 36;
    if (code < 58) return true;
    if (code < 65) return false;
    if (code < 91) return true;
    if (code < 97) return code === 95;
    if (code < 123)return true;
    return code >= 0xaa && nonASCIIidentifier.test(String.fromCharCode(code));
  };

  // ## Tokenizer

  // These are used when `options.locations` is on, for the
  // `tokStartLoc` and `tokEndLoc` properties.

  function line_loc_t() {
    this.line = tokCurLine;
    this.column = tokPos - tokLineStart;
  }

  // Reset the token state. Used at the start of a parse.

  function initTokenState() {
    tokCurLine = 1;
    tokPos = tokLineStart = 0;
    tokRegexpAllowed = true;
    skipSpace();
  }

  // Called at the end of every token. Sets `tokEnd`, `tokVal`, and
  // `tokRegexpAllowed`, and skips the space after the token, so that
  // the next one's `tokStart` will point at the right position.

  function finishToken(type, val) {
    tokEnd = tokPos;
    if (options.locations) tokEndLoc = new line_loc_t;
    tokType = type;
    skipSpace();
    tokVal = val;
    tokRegexpAllowed = type.beforeExpr;
  }

  function skipBlockComment() {
    var startLoc = options.onComment && options.locations && new line_loc_t;
    var start = tokPos, end = input.indexOf("*/", tokPos += 2);
    if (end === -1) raise(tokPos - 2, "Unterminated comment");
    tokPos = end + 2;
    if (options.locations) {
      lineBreak.lastIndex = start;
      var match;
      while ((match = lineBreak.exec(input)) && match.index < tokPos) {
        ++tokCurLine;
        tokLineStart = match.index + match[0].length;
      }
    }
    if (options.onComment)
      options.onComment(true, input.slice(start + 2, end), start, tokPos,
                        startLoc, options.locations && new line_loc_t);
  }

  function skipLineComment() {
    var start = tokPos;
    var startLoc = options.onComment && options.locations && new line_loc_t;
    var ch = input.charCodeAt(tokPos+=2);
    while (tokPos < inputLen && ch !== 10 && ch !== 13 && ch !== 8232 && ch !== 8233) {
      ++tokPos;
      ch = input.charCodeAt(tokPos);
    }
    if (options.onComment)
      options.onComment(false, input.slice(start + 2, tokPos), start, tokPos,
                        startLoc, options.locations && new line_loc_t);
  }

  // Called at the start of the parse and after every token. Skips
  // whitespace and comments, and.

  function skipSpace() {
    while (tokPos < inputLen) {
      var ch = input.charCodeAt(tokPos);
      if (ch === 32) { // ' '
        ++tokPos;
      } else if (ch === 13) {
        ++tokPos;
        var next = input.charCodeAt(tokPos);
        if (next === 10) {
          ++tokPos;
        }
        if (options.locations) {
          ++tokCurLine;
          tokLineStart = tokPos;
        }
      } else if (ch === 10 || ch === 8232 || ch === 8233) {
        ++tokPos;
        if (options.locations) {
          ++tokCurLine;
          tokLineStart = tokPos;
        }
      } else if (ch > 8 && ch < 14) {
        ++tokPos;
      } else if (ch === 47) { // '/'
        var next = input.charCodeAt(tokPos + 1);
        if (next === 42) { // '*'
          skipBlockComment();
        } else if (next === 47) { // '/'
          skipLineComment();
        } else break;
      } else if (ch === 160) { // '\xa0'
        ++tokPos;
      } else if (ch >= 5760 && nonASCIIwhitespace.test(String.fromCharCode(ch))) {
        ++tokPos;
      } else {
        break;
      }
    }
  }

  // ### Token reading

  // This is the function that is called to fetch the next token. It
  // is somewhat obscure, because it works in character codes rather
  // than characters, and because operator parsing has been inlined
  // into it.
  //
  // All in the name of speed.
  //
  // The `forceRegexp` parameter is used in the one case where the
  // `tokRegexpAllowed` trick does not work. See `parseStatement`.

  function readToken_dot() {
    var next = input.charCodeAt(tokPos + 1);
    if (next >= 48 && next <= 57) return readNumber(true);
    ++tokPos;
    return finishToken(_dot);
  }

  function readToken_slash() { // '/'
    var next = input.charCodeAt(tokPos + 1);
    if (tokRegexpAllowed) {++tokPos; return readRegexp();}
    if (next === 61) return finishOp(_assign, 2);
    return finishOp(_slash, 1);
  }

  function readToken_mult_modulo() { // '%*'
    var next = input.charCodeAt(tokPos + 1);
    if (next === 61) return finishOp(_assign, 2);
    return finishOp(_multiplyModulo, 1);
  }

  function readToken_pipe_amp(code) { // '|&'
    var next = input.charCodeAt(tokPos + 1);
    if (next === code) return finishOp(code === 124 ? _logicalOR : _logicalAND, 2);
    if (next === 61) return finishOp(_assign, 2);
    return finishOp(code === 124 ? _bitwiseOR : _bitwiseAND, 1);
  }

  function readToken_caret() { // '^'
    var next = input.charCodeAt(tokPos + 1);
    if (next === 61) return finishOp(_assign, 2);
    return finishOp(_bitwiseXOR, 1);
  }

  function readToken_plus_min(code) { // '+-'
    var next = input.charCodeAt(tokPos + 1);
    if (next === code) {
      if (next == 45 && input.charCodeAt(tokPos + 2) == 62 &&
          newline.test(input.slice(lastEnd, tokPos))) {
        // A `-->` line comment
        tokPos += 3;
        skipLineComment();
        skipSpace();
        return readToken();
      }
      return finishOp(_incDec, 2);
    }
    if (next === 61) return finishOp(_assign, 2);
    return finishOp(_plusMin, 1);
  }

  function readToken_lt_gt(code) { // '<>'
    var next = input.charCodeAt(tokPos + 1);
    var size = 1;
    if (next === code) {
      size = code === 62 && input.charCodeAt(tokPos + 2) === 62 ? 3 : 2;
      if (input.charCodeAt(tokPos + size) === 61) return finishOp(_assign, size + 1);
      return finishOp(_bitShift, size);
    }
    if (next == 33 && code == 60 && input.charCodeAt(tokPos + 2) == 45 &&
        input.charCodeAt(tokPos + 3) == 45) {
      // `<!--`, an XML-style comment that should be interpreted as a line comment
      tokPos += 4;
      skipLineComment();
      skipSpace();
      return readToken();
    }
    if (next === 61)
      size = input.charCodeAt(tokPos + 2) === 61 ? 3 : 2;
    return finishOp(_relational, size);
  }

  function readToken_eq_excl(code) { // '=!'
    var next = input.charCodeAt(tokPos + 1);
    if (next === 61) return finishOp(_equality, input.charCodeAt(tokPos + 2) === 61 ? 3 : 2);
    return finishOp(code === 61 ? _eq : _prefix, 1);
  }

  function getTokenFromCode(code) {
    switch(code) {
      // The interpretation of a dot depends on whether it is followed
      // by a digit.
    case 46: // '.'
      return readToken_dot();

      // Punctuation tokens.
    case 40: ++tokPos; return finishToken(_parenL);
    case 41: ++tokPos; return finishToken(_parenR);
    case 59: ++tokPos; return finishToken(_semi);
    case 44: ++tokPos; return finishToken(_comma);
    case 91: ++tokPos; return finishToken(_bracketL);
    case 93: ++tokPos; return finishToken(_bracketR);
    case 123: ++tokPos; return finishToken(_braceL);
    case 125: ++tokPos; return finishToken(_braceR);
    case 58: ++tokPos; return finishToken(_colon);
    case 63: ++tokPos; return finishToken(_question);

      // '0x' is a hexadecimal number.
    case 48: // '0'
      var next = input.charCodeAt(tokPos + 1);
      if (next === 120 || next === 88) return readHexNumber();
      // Anything else beginning with a digit is an integer, octal
      // number, or float.
    case 49: case 50: case 51: case 52: case 53: case 54: case 55: case 56: case 57: // 1-9
      return readNumber(false);

      // Quotes produce strings.
    case 34: case 39: // '"', "'"
      return readString(code);

    // Operators are parsed inline in tiny state machines. '=' (61) is
    // often referred to. `finishOp` simply skips the amount of
    // characters it is given as second argument, and returns a token
    // of the type given by its first argument.

    case 47: // '/'
      return readToken_slash(code);

    case 37: case 42: // '%*'
      return readToken_mult_modulo();

    case 124: case 38: // '|&'
      return readToken_pipe_amp(code);

    case 94: // '^'
      return readToken_caret();

    case 43: case 45: // '+-'
      return readToken_plus_min(code);

    case 60: case 62: // '<>'
      return readToken_lt_gt(code);

    case 61: case 33: // '=!'
      return readToken_eq_excl(code);

    case 126: // '~'
      return finishOp(_prefix, 1);
    }

    return false;
  }

  function readToken(forceRegexp) {
    if (!forceRegexp) tokStart = tokPos;
    else tokPos = tokStart + 1;
    if (options.locations) tokStartLoc = new line_loc_t;
    if (forceRegexp) return readRegexp();
    if (tokPos >= inputLen) return finishToken(_eof);

    var code = input.charCodeAt(tokPos);
    // Identifier or keyword. '\uXXXX' sequences are allowed in
    // identifiers, so '\' also dispatches to that.
    if (isIdentifierStart(code) || code === 92 /* '\' */) return readWord();

    var tok = getTokenFromCode(code);

    if (tok === false) {
      // If we are here, we either found a non-ASCII identifier
      // character, or something that's entirely disallowed.
      var ch = String.fromCharCode(code);
      if (ch === "\\" || nonASCIIidentifierStart.test(ch)) return readWord();
      raise(tokPos, "Unexpected character '" + ch + "'");
    }
    return tok;
  }

  function finishOp(type, size) {
    var str = input.slice(tokPos, tokPos + size);
    tokPos += size;
    finishToken(type, str);
  }

  // Parse a regular expression. Some context-awareness is necessary,
  // since a '/' inside a '[]' set does not end the expression.

  function readRegexp() {
    var content = "", escaped, inClass, start = tokPos;
    for (;;) {
      if (tokPos >= inputLen) raise(start, "Unterminated regular expression");
      var ch = input.charAt(tokPos);
      if (newline.test(ch)) raise(start, "Unterminated regular expression");
      if (!escaped) {
        if (ch === "[") inClass = true;
        else if (ch === "]" && inClass) inClass = false;
        else if (ch === "/" && !inClass) break;
        escaped = ch === "\\";
      } else escaped = false;
      ++tokPos;
    }
    var content = input.slice(start, tokPos);
    ++tokPos;
    // Need to use `readWord1` because '\uXXXX' sequences are allowed
    // here (don't ask).
    var mods = readWord1();
    if (mods && !/^[gmsiy]*$/.test(mods)) raise(start, "Invalid regexp flag");
    return finishToken(_regexp, new RegExp(content, mods));
  }

  // Read an integer in the given radix. Return null if zero digits
  // were read, the integer value otherwise. When `len` is given, this
  // will return `null` unless the integer has exactly `len` digits.

  function readInt(radix, len) {
    var start = tokPos, total = 0;
    for (var i = 0, e = len == null ? Infinity : len; i < e; ++i) {
      var code = input.charCodeAt(tokPos), val;
      if (code >= 97) val = code - 97 + 10; // a
      else if (code >= 65) val = code - 65 + 10; // A
      else if (code >= 48 && code <= 57) val = code - 48; // 0-9
      else val = Infinity;
      if (val >= radix) break;
      ++tokPos;
      total = total * radix + val;
    }
    if (tokPos === start || len != null && tokPos - start !== len) return null;

    return total;
  }

  function readHexNumber() {
    tokPos += 2; // 0x
    var val = readInt(16);
    if (val == null) raise(tokStart + 2, "Expected hexadecimal number");
    if (isIdentifierStart(input.charCodeAt(tokPos))) raise(tokPos, "Identifier directly after number");
    return finishToken(_num, val);
  }

  // Read an integer, octal integer, or floating-point number.

  function readNumber(startsWithDot) {
    var start = tokPos, isFloat = false, octal = input.charCodeAt(tokPos) === 48;
    if (!startsWithDot && readInt(10) === null) raise(start, "Invalid number");
    if (input.charCodeAt(tokPos) === 46) {
      ++tokPos;
      readInt(10);
      isFloat = true;
    }
    var next = input.charCodeAt(tokPos);
    if (next === 69 || next === 101) { // 'eE'
      next = input.charCodeAt(++tokPos);
      if (next === 43 || next === 45) ++tokPos; // '+-'
      if (readInt(10) === null) raise(start, "Invalid number");
      isFloat = true;
    }
    if (isIdentifierStart(input.charCodeAt(tokPos))) raise(tokPos, "Identifier directly after number");

    var str = input.slice(start, tokPos), val;
    if (isFloat) val = parseFloat(str);
    else if (!octal || str.length === 1) val = parseInt(str, 10);
    else if (/[89]/.test(str) || strict) raise(start, "Invalid number");
    else val = parseInt(str, 8);
    return finishToken(_num, val);
  }

  // Read a string value, interpreting backslash-escapes.

  function readString(quote) {
    tokPos++;
    var out = "";
    for (;;) {
      if (tokPos >= inputLen) raise(tokStart, "Unterminated string constant");
      var ch = input.charCodeAt(tokPos);
      if (ch === quote) {
        ++tokPos;
        return finishToken(_string, out);
      }
      if (ch === 92) { // '\'
        ch = input.charCodeAt(++tokPos);
        var octal = /^[0-7]+/.exec(input.slice(tokPos, tokPos + 3));
        if (octal) octal = octal[0];
        while (octal && parseInt(octal, 8) > 255) octal = octal.slice(0, -1);
        if (octal === "0") octal = null;
        ++tokPos;
        if (octal) {
          if (strict) raise(tokPos - 2, "Octal literal in strict mode");
          out += String.fromCharCode(parseInt(octal, 8));
          tokPos += octal.length - 1;
        } else {
          switch (ch) {
          case 110: out += "\n"; break; // 'n' -> '\n'
          case 114: out += "\r"; break; // 'r' -> '\r'
          case 120: out += String.fromCharCode(readHexChar(2)); break; // 'x'
          case 117: out += String.fromCharCode(readHexChar(4)); break; // 'u'
          case 85: out += String.fromCharCode(readHexChar(8)); break; // 'U'
          case 116: out += "\t"; break; // 't' -> '\t'
          case 98: out += "\b"; break; // 'b' -> '\b'
          case 118: out += "\u000b"; break; // 'v' -> '\u000b'
          case 102: out += "\f"; break; // 'f' -> '\f'
          case 48: out += "\0"; break; // 0 -> '\0'
          case 13: if (input.charCodeAt(tokPos) === 10) ++tokPos; // '\r\n'
          case 10: // ' \n'
            if (options.locations) { tokLineStart = tokPos; ++tokCurLine; }
            break;
          default: out += String.fromCharCode(ch); break;
          }
        }
      } else {
        if (ch === 13 || ch === 10 || ch === 8232 || ch === 8233) raise(tokStart, "Unterminated string constant");
        out += String.fromCharCode(ch); // '\'
        ++tokPos;
      }
    }
  }

  // Used to read character escape sequences ('\x', '\u', '\U').

  function readHexChar(len) {
    var n = readInt(16, len);
    if (n === null) raise(tokStart, "Bad character escape sequence");
    return n;
  }

  // Used to signal to callers of `readWord1` whether the word
  // contained any escape sequences. This is needed because words with
  // escape sequences must not be interpreted as keywords.

  var containsEsc;

  // Read an identifier, and return it as a string. Sets `containsEsc`
  // to whether the word contained a '\u' escape.
  //
  // Only builds up the word character-by-character when it actually
  // containeds an escape, as a micro-optimization.

  function readWord1() {
    containsEsc = false;
    var word, first = true, start = tokPos;
    for (;;) {
      var ch = input.charCodeAt(tokPos);
      if (isIdentifierChar(ch)) {
        if (containsEsc) word += input.charAt(tokPos);
        ++tokPos;
      } else if (ch === 92) { // "\"
        if (!containsEsc) word = input.slice(start, tokPos);
        containsEsc = true;
        if (input.charCodeAt(++tokPos) != 117) // "u"
          raise(tokPos, "Expecting Unicode escape sequence \\uXXXX");
        ++tokPos;
        var esc = readHexChar(4);
        var escStr = String.fromCharCode(esc);
        if (!escStr) raise(tokPos - 1, "Invalid Unicode escape");
        if (!(first ? isIdentifierStart(esc) : isIdentifierChar(esc)))
          raise(tokPos - 4, "Invalid Unicode escape");
        word += escStr;
      } else {
        break;
      }
      first = false;
    }
    return containsEsc ? word : input.slice(start, tokPos);
  }

  // Read an identifier or keyword token. Will check for reserved
  // words when necessary.

  function readWord() {
    var word = readWord1();
    var type = _name;
    if (!containsEsc) {
      if (isKeyword(word)) type = keywordTypes[word];
      else if (options.forbidReserved &&
               (options.ecmaVersion === 3 ? isReservedWord3 : isReservedWord5)(word) ||
               strict && isStrictReservedWord(word))
        raise(tokStart, "The keyword '" + word + "' is reserved");
    }
    return finishToken(type, word);
  }

  // ## Parser

  // A recursive descent parser operates by defining functions for all
  // syntactic elements, and recursively calling those, each function
  // advancing the input stream and returning an AST node. Precedence
  // of constructs (for example, the fact that `!x[1]` means `!(x[1])`
  // instead of `(!x)[1]` is handled by the fact that the parser
  // function that parses unary prefix operators is called first, and
  // in turn calls the function that parses `[]` subscripts — that
  // way, it'll receive the node for `x[1]` already parsed, and wraps
  // *that* in the unary operator node.
  //
  // Acorn uses an [operator precedence parser][opp] to handle binary
  // operator precedence, because it is much more compact than using
  // the technique outlined above, which uses different, nesting
  // functions to specify precedence, for all of the ten binary
  // precedence levels that JavaScript defines.
  //
  // [opp]: http://en.wikipedia.org/wiki/Operator-precedence_parser

  // ### Parser utilities

  // Continue to the next token.

  function next() {
    lastStart = tokStart;
    lastEnd = tokEnd;
    lastEndLoc = tokEndLoc;
    readToken();
  }

  // Enter strict mode. Re-reads the next token to please pedantic
  // tests ("use strict"; 010; -- should fail).

  function setStrict(strct) {
    strict = strct;
    tokPos = lastEnd;
    if (options.locations) {
      while (tokPos < tokLineStart) {
        tokLineStart = input.lastIndexOf("\n", tokLineStart - 2) + 1;
        --tokCurLine;
      }
    }
    skipSpace();
    readToken();
  }

  // Start an AST node, attaching a start offset.

  function node_t() {
    this.type = null;
    this.start = tokStart;
    this.end = null;
  }

  function node_loc_t() {
    this.start = tokStartLoc;
    this.end = null;
    if (sourceFile !== null) this.source = sourceFile;
  }

  function startNode() {
    var node = new node_t();
    if (options.locations)
      node.loc = new node_loc_t();
    if (options.directSourceFile)
      node.sourceFile = options.directSourceFile;
    if (options.ranges)
      node.range = [tokStart, 0];
    return node;
  }

  // Start a node whose start offset information should be based on
  // the start of another node. For example, a binary operator node is
  // only started after its left-hand side has already been parsed.

  function startNodeFrom(other) {
    var node = new node_t();
    node.start = other.start;
    if (options.locations) {
      node.loc = new node_loc_t();
      node.loc.start = other.loc.start;
    }
    if (options.ranges)
      node.range = [other.range[0], 0];

    return node;
  }

  // Finish an AST node, adding `type` and `end` properties.

  function finishNode(node, type) {
    node.type = type;
    node.end = lastEnd;
    if (options.locations)
      node.loc.end = lastEndLoc;
    if (options.ranges)
      node.range[1] = lastEnd;
    return node;
  }

  // Test whether a statement node is the string literal `"use strict"`.

  function isUseStrict(stmt) {
    return options.ecmaVersion >= 5 && stmt.type === "ExpressionStatement" &&
      stmt.expression.type === "Literal" && stmt.expression.value === "use strict";
  }

  // Predicate that tests whether the next token is of the given
  // type, and if yes, consumes it as a side effect.

  function eat(type) {
    if (tokType === type) {
      next();
      return true;
    }
  }

  // Test whether a semicolon can be inserted at the current position.

  function canInsertSemicolon() {
    return !options.strictSemicolons &&
      (tokType === _eof || tokType === _braceR || newline.test(input.slice(lastEnd, tokStart)));
  }

  // Consume a semicolon, or, failing that, see if we are allowed to
  // pretend that there is a semicolon at this position.

  function semicolon() {
    if (!eat(_semi) && !canInsertSemicolon()) unexpected();
  }

  // Expect a token of a given type. If found, consume it, otherwise,
  // raise an unexpected token error.

  function expect(type) {
    if (tokType === type) next();
    else unexpected();
  }

  // Raise an unexpected token error.

  function unexpected() {
    raise(tokStart, "Unexpected token");
  }

  // Verify that a node is an lval — something that can be assigned
  // to.

  function checkLVal(expr) {
    if (expr.type !== "Identifier" && expr.type !== "MemberExpression")
      raise(expr.start, "Assigning to rvalue");
    if (strict && expr.type === "Identifier" && isStrictBadIdWord(expr.name))
      raise(expr.start, "Assigning to " + expr.name + " in strict mode");
  }

  // ### Statement parsing

  // Parse a program. Initializes the parser, reads any number of
  // statements, and wraps them in a Program node.  Optionally takes a
  // `program` argument.  If present, the statements will be appended
  // to its body instead of creating a new node.

  function parseTopLevel(program) {
    lastStart = lastEnd = tokPos;
    if (options.locations) lastEndLoc = new line_loc_t;
    inFunction = strict = null;
    labels = [];
    readToken();

    var node = program || startNode(), first = true;
    if (!program) node.body = [];
    while (tokType !== _eof) {
      var stmt = parseStatement();
      node.body.push(stmt);
      if (first && isUseStrict(stmt)) setStrict(true);
      first = false;
    }
    return finishNode(node, "Program");
  }

  var loopLabel = {kind: "loop"}, switchLabel = {kind: "switch"};

  // Parse a single statement.
  //
  // If expecting a statement and finding a slash operator, parse a
  // regular expression literal. This is to handle cases like
  // `if (foo) /blah/.exec(foo);`, where looking at the previous token
  // does not help.

  function parseStatement() {
    if (tokType === _slash || tokType === _assign && tokVal == "/=")
      readToken(true);

    var starttype = tokType, node = startNode();

    // Most types of statements are recognized by the keyword they
    // start with. Many are trivial to parse, some require a bit of
    // complexity.

    switch (starttype) {
    case _break: case _continue:
      next();
      var isBreak = starttype === _break;
      if (eat(_semi) || canInsertSemicolon()) node.label = null;
      else if (tokType !== _name) unexpected();
      else {
        node.label = parseIdent();
        semicolon();
      }

      // Verify that there is an actual destination to break or
      // continue to.
      for (var i = 0; i < labels.length; ++i) {
        var lab = labels[i];
        if (node.label == null || lab.name === node.label.name) {
          if (lab.kind != null && (isBreak || lab.kind === "loop")) break;
          if (node.label && isBreak) break;
        }
      }
      if (i === labels.length) raise(node.start, "Unsyntactic " + starttype.keyword);
      return finishNode(node, isBreak ? "BreakStatement" : "ContinueStatement");

    case _debugger:
      next();
      semicolon();
      return finishNode(node, "DebuggerStatement");

    case _do:
      next();
      labels.push(loopLabel);
      node.body = parseStatement();
      labels.pop();
      expect(_while);
      node.test = parseParenExpression();
      semicolon();
      return finishNode(node, "DoWhileStatement");

      // Disambiguating between a `for` and a `for`/`in` loop is
      // non-trivial. Basically, we have to parse the init `var`
      // statement or expression, disallowing the `in` operator (see
      // the second parameter to `parseExpression`), and then check
      // whether the next token is `in`. When there is no init part
      // (semicolon immediately after the opening parenthesis), it is
      // a regular `for` loop.

    case _for:
      next();
      labels.push(loopLabel);
      expect(_parenL);
      if (tokType === _semi) return parseFor(node, null);
      if (tokType === _var) {
        var init = startNode();
        next();
        parseVar(init, true);
        finishNode(init, "VariableDeclaration");
        if (init.declarations.length === 1 && eat(_in))
          return parseForIn(node, init);
        return parseFor(node, init);
      }
      var init = parseExpression(false, true);
      if (eat(_in)) {checkLVal(init); return parseForIn(node, init);}
      return parseFor(node, init);

    case _function:
      next();
      return parseFunction(node, true);

    case _if:
      next();
      node.test = parseParenExpression();
      node.consequent = parseStatement();
      node.alternate = eat(_else) ? parseStatement() : null;
      return finishNode(node, "IfStatement");

    case _return:
      if (!inFunction) raise(tokStart, "'return' outside of function");
      next();

      // In `return` (and `break`/`continue`), the keywords with
      // optional arguments, we eagerly look for a semicolon or the
      // possibility to insert one.

      if (eat(_semi) || canInsertSemicolon()) node.argument = null;
      else { node.argument = parseExpression(); semicolon(); }
      return finishNode(node, "ReturnStatement");

    case _switch:
      next();
      node.discriminant = parseParenExpression();
      node.cases = [];
      expect(_braceL);
      labels.push(switchLabel);

      // Statements under must be grouped (by label) in SwitchCase
      // nodes. `cur` is used to keep the node that we are currently
      // adding statements to.

      for (var cur, sawDefault; tokType != _braceR;) {
        if (tokType === _case || tokType === _default) {
          var isCase = tokType === _case;
          if (cur) finishNode(cur, "SwitchCase");
          node.cases.push(cur = startNode());
          cur.consequent = [];
          next();
          if (isCase) cur.test = parseExpression();
          else {
            if (sawDefault) raise(lastStart, "Multiple default clauses"); sawDefault = true;
            cur.test = null;
          }
          expect(_colon);
        } else {
          if (!cur) unexpected();
          cur.consequent.push(parseStatement());
        }
      }
      if (cur) finishNode(cur, "SwitchCase");
      next(); // Closing brace
      labels.pop();
      return finishNode(node, "SwitchStatement");

    case _throw:
      next();
      if (newline.test(input.slice(lastEnd, tokStart)))
        raise(lastEnd, "Illegal newline after throw");
      node.argument = parseExpression();
      semicolon();
      return finishNode(node, "ThrowStatement");

    case _try:
      next();
      node.block = parseBlock();
      node.handler = null;
      if (tokType === _catch) {
        var clause = startNode();
        next();
        expect(_parenL);
        clause.param = parseIdent();
        if (strict && isStrictBadIdWord(clause.param.name))
          raise(clause.param.start, "Binding " + clause.param.name + " in strict mode");
        expect(_parenR);
        clause.guard = null;
        clause.body = parseBlock();
        node.handler = finishNode(clause, "CatchClause");
      }
      node.guardedHandlers = empty;
      node.finalizer = eat(_finally) ? parseBlock() : null;
      if (!node.handler && !node.finalizer)
        raise(node.start, "Missing catch or finally clause");
      return finishNode(node, "TryStatement");

    case _var:
      next();
      parseVar(node);
      semicolon();
      return finishNode(node, "VariableDeclaration");

    case _while:
      next();
      node.test = parseParenExpression();
      labels.push(loopLabel);
      node.body = parseStatement();
      labels.pop();
      return finishNode(node, "WhileStatement");

    case _with:
      if (strict) raise(tokStart, "'with' in strict mode");
      next();
      node.object = parseParenExpression();
      node.body = parseStatement();
      return finishNode(node, "WithStatement");

    case _braceL:
      return parseBlock();

    case _semi:
      next();
      return finishNode(node, "EmptyStatement");

      // If the statement does not start with a statement keyword or a
      // brace, it's an ExpressionStatement or LabeledStatement. We
      // simply start parsing an expression, and afterwards, if the
      // next token is a colon and the expression was a simple
      // Identifier node, we switch to interpreting it as a label.

    default:
      var maybeName = tokVal, expr = parseExpression();
      if (starttype === _name && expr.type === "Identifier" && eat(_colon)) {
        for (var i = 0; i < labels.length; ++i)
          if (labels[i].name === maybeName) raise(expr.start, "Label '" + maybeName + "' is already declared");
        var kind = tokType.isLoop ? "loop" : tokType === _switch ? "switch" : null;
        labels.push({name: maybeName, kind: kind});
        node.body = parseStatement();
        labels.pop();
        node.label = expr;
        return finishNode(node, "LabeledStatement");
      } else {
        node.expression = expr;
        semicolon();
        return finishNode(node, "ExpressionStatement");
      }
    }
  }

  // Used for constructs like `switch` and `if` that insist on
  // parentheses around their expression.

  function parseParenExpression() {
    expect(_parenL);
    var val = parseExpression();
    expect(_parenR);
    return val;
  }

  // Parse a semicolon-enclosed block of statements, handling `"use
  // strict"` declarations when `allowStrict` is true (used for
  // function bodies).

  function parseBlock(allowStrict) {
    var node = startNode(), first = true, strict = false, oldStrict;
    node.body = [];
    expect(_braceL);
    while (!eat(_braceR)) {
      var stmt = parseStatement();
      node.body.push(stmt);
      if (first && allowStrict && isUseStrict(stmt)) {
        oldStrict = strict;
        setStrict(strict = true);
      }
      first = false;
    }
    if (strict && !oldStrict) setStrict(false);
    return finishNode(node, "BlockStatement");
  }

  // Parse a regular `for` loop. The disambiguation code in
  // `parseStatement` will already have parsed the init statement or
  // expression.

  function parseFor(node, init) {
    node.init = init;
    expect(_semi);
    node.test = tokType === _semi ? null : parseExpression();
    expect(_semi);
    node.update = tokType === _parenR ? null : parseExpression();
    expect(_parenR);
    node.body = parseStatement();
    labels.pop();
    return finishNode(node, "ForStatement");
  }

  // Parse a `for`/`in` loop.

  function parseForIn(node, init) {
    node.left = init;
    node.right = parseExpression();
    expect(_parenR);
    node.body = parseStatement();
    labels.pop();
    return finishNode(node, "ForInStatement");
  }

  // Parse a list of variable declarations.

  function parseVar(node, noIn) {
    node.declarations = [];
    node.kind = "var";
    for (;;) {
      var decl = startNode();
      decl.id = parseIdent();
      if (strict && isStrictBadIdWord(decl.id.name))
        raise(decl.id.start, "Binding " + decl.id.name + " in strict mode");
      decl.init = eat(_eq) ? parseExpression(true, noIn) : null;
      node.declarations.push(finishNode(decl, "VariableDeclarator"));
      if (!eat(_comma)) break;
    }
    return node;
  }

  // ### Expression parsing

  // These nest, from the most general expression type at the top to
  // 'atomic', nondivisible expression types at the bottom. Most of
  // the functions will simply let the function(s) below them parse,
  // and, *if* the syntactic construct they handle is present, wrap
  // the AST node that the inner parser gave them in another node.

  // Parse a full expression. The arguments are used to forbid comma
  // sequences (in argument lists, array literals, or object literals)
  // or the `in` operator (in for loops initalization expressions).

  function parseExpression(noComma, noIn) {
    var expr = parseMaybeAssign(noIn);
    if (!noComma && tokType === _comma) {
      var node = startNodeFrom(expr);
      node.expressions = [expr];
      while (eat(_comma)) node.expressions.push(parseMaybeAssign(noIn));
      return finishNode(node, "SequenceExpression");
    }
    return expr;
  }

  // Parse an assignment expression. This includes applications of
  // operators like `+=`.

  function parseMaybeAssign(noIn) {
    var left = parseMaybeConditional(noIn);
    if (tokType.isAssign) {
      var node = startNodeFrom(left);
      node.operator = tokVal;
      node.left = left;
      next();
      node.right = parseMaybeAssign(noIn);
      checkLVal(left);
      return finishNode(node, "AssignmentExpression");
    }
    return left;
  }

  // Parse a ternary conditional (`?:`) operator.

  function parseMaybeConditional(noIn) {
    var expr = parseExprOps(noIn);
    if (eat(_question)) {
      var node = startNodeFrom(expr);
      node.test = expr;
      node.consequent = parseExpression(true);
      expect(_colon);
      node.alternate = parseExpression(true, noIn);
      return finishNode(node, "ConditionalExpression");
    }
    return expr;
  }

  // Start the precedence parser.

  function parseExprOps(noIn) {
    return parseExprOp(parseMaybeUnary(), -1, noIn);
  }

  // Parse binary operators with the operator precedence parsing
  // algorithm. `left` is the left-hand side of the operator.
  // `minPrec` provides context that allows the function to stop and
  // defer further parser to one of its callers when it encounters an
  // operator that has a lower precedence than the set it is parsing.

  function parseExprOp(left, minPrec, noIn) {
    var prec = tokType.binop;
    if (prec != null && (!noIn || tokType !== _in)) {
      if (prec > minPrec) {
        var node = startNodeFrom(left);
        node.left = left;
        node.operator = tokVal;
        var op = tokType;
        next();
        node.right = parseExprOp(parseMaybeUnary(), prec, noIn);
        var exprNode = finishNode(node, (op === _logicalOR || op === _logicalAND) ? "LogicalExpression" : "BinaryExpression");
        return parseExprOp(exprNode, minPrec, noIn);
      }
    }
    return left;
  }

  // Parse unary operators, both prefix and postfix.

  function parseMaybeUnary() {
    if (tokType.prefix) {
      var node = startNode(), update = tokType.isUpdate;
      node.operator = tokVal;
      node.prefix = true;
      tokRegexpAllowed = true;
      next();
      node.argument = parseMaybeUnary();
      if (update) checkLVal(node.argument);
      else if (strict && node.operator === "delete" &&
               node.argument.type === "Identifier")
        raise(node.start, "Deleting local variable in strict mode");
      return finishNode(node, update ? "UpdateExpression" : "UnaryExpression");
    }
    var expr = parseExprSubscripts();
    while (tokType.postfix && !canInsertSemicolon()) {
      var node = startNodeFrom(expr);
      node.operator = tokVal;
      node.prefix = false;
      node.argument = expr;
      checkLVal(expr);
      next();
      expr = finishNode(node, "UpdateExpression");
    }
    return expr;
  }

  // Parse call, dot, and `[]`-subscript expressions.

  function parseExprSubscripts() {
    return parseSubscripts(parseExprAtom());
  }

  function parseSubscripts(base, noCalls) {
    if (eat(_dot)) {
      var node = startNodeFrom(base);
      node.object = base;
      node.property = parseIdent(true);
      node.computed = false;
      return parseSubscripts(finishNode(node, "MemberExpression"), noCalls);
    } else if (eat(_bracketL)) {
      var node = startNodeFrom(base);
      node.object = base;
      node.property = parseExpression();
      node.computed = true;
      expect(_bracketR);
      return parseSubscripts(finishNode(node, "MemberExpression"), noCalls);
    } else if (!noCalls && eat(_parenL)) {
      var node = startNodeFrom(base);
      node.callee = base;
      node.arguments = parseExprList(_parenR, false);
      return parseSubscripts(finishNode(node, "CallExpression"), noCalls);
    } else return base;
  }

  // Parse an atomic expression — either a single token that is an
  // expression, an expression started by a keyword like `function` or
  // `new`, or an expression wrapped in punctuation like `()`, `[]`,
  // or `{}`.

  function parseExprAtom() {
    switch (tokType) {
    case _this:
      var node = startNode();
      next();
      return finishNode(node, "ThisExpression");
    case _name:
      return parseIdent();
    case _num: case _string: case _regexp:
      var node = startNode();
      node.value = tokVal;
      node.raw = input.slice(tokStart, tokEnd);
      next();
      return finishNode(node, "Literal");

    case _null: case _true: case _false:
      var node = startNode();
      node.value = tokType.atomValue;
      node.raw = tokType.keyword;
      next();
      return finishNode(node, "Literal");

    case _parenL:
      var tokStartLoc1 = tokStartLoc, tokStart1 = tokStart;
      next();
      var val = parseExpression();
      val.start = tokStart1;
      val.end = tokEnd;
      if (options.locations) {
        val.loc.start = tokStartLoc1;
        val.loc.end = tokEndLoc;
      }
      if (options.ranges)
        val.range = [tokStart1, tokEnd];
      expect(_parenR);
      return val;

    case _bracketL:
      var node = startNode();
      next();
      node.elements = parseExprList(_bracketR, true, true);
      return finishNode(node, "ArrayExpression");

    case _braceL:
      return parseObj();

    case _function:
      var node = startNode();
      next();
      return parseFunction(node, false);

    case _new:
      return parseNew();

    default:
      unexpected();
    }
  }

  // New's precedence is slightly tricky. It must allow its argument
  // to be a `[]` or dot subscript expression, but not a call — at
  // least, not without wrapping it in parentheses. Thus, it uses the

  function parseNew() {
    var node = startNode();
    next();
    node.callee = parseSubscripts(parseExprAtom(), true);
    if (eat(_parenL)) node.arguments = parseExprList(_parenR, false);
    else node.arguments = empty;
    return finishNode(node, "NewExpression");
  }

  // Parse an object literal.

  function parseObj() {
    var node = startNode(), first = true, sawGetSet = false;
    node.properties = [];
    next();
    while (!eat(_braceR)) {
      if (!first) {
        expect(_comma);
        if (options.allowTrailingCommas && eat(_braceR)) break;
      } else first = false;

      var prop = {key: parsePropertyName()}, isGetSet = false, kind;
      if (eat(_colon)) {
        prop.value = parseExpression(true);
        kind = prop.kind = "init";
      } else if (options.ecmaVersion >= 5 && prop.key.type === "Identifier" &&
                 (prop.key.name === "get" || prop.key.name === "set")) {
        isGetSet = sawGetSet = true;
        kind = prop.kind = prop.key.name;
        prop.key = parsePropertyName();
        if (tokType !== _parenL) unexpected();
        prop.value = parseFunction(startNode(), false);
      } else unexpected();

      // getters and setters are not allowed to clash — either with
      // each other or with an init property — and in strict mode,
      // init properties are also not allowed to be repeated.

      if (prop.key.type === "Identifier" && (strict || sawGetSet)) {
        for (var i = 0; i < node.properties.length; ++i) {
          var other = node.properties[i];
          if (other.key.name === prop.key.name) {
            var conflict = kind == other.kind || isGetSet && other.kind === "init" ||
              kind === "init" && (other.kind === "get" || other.kind === "set");
            if (conflict && !strict && kind === "init" && other.kind === "init") conflict = false;
            if (conflict) raise(prop.key.start, "Redefinition of property");
          }
        }
      }
      node.properties.push(prop);
    }
    return finishNode(node, "ObjectExpression");
  }

  function parsePropertyName() {
    if (tokType === _num || tokType === _string) return parseExprAtom();
    return parseIdent(true);
  }

  // Parse a function declaration or literal (depending on the
  // `isStatement` parameter).

  function parseFunction(node, isStatement) {
    if (tokType === _name) node.id = parseIdent();
    else if (isStatement) unexpected();
    else node.id = null;
    node.params = [];
    var first = true;
    expect(_parenL);
    while (!eat(_parenR)) {
      if (!first) expect(_comma); else first = false;
      node.params.push(parseIdent());
    }

    // Start a new scope with regard to labels and the `inFunction`
    // flag (restore them to their old value afterwards).
    var oldInFunc = inFunction, oldLabels = labels;
    inFunction = true; labels = [];
    node.body = parseBlock(true);
    inFunction = oldInFunc; labels = oldLabels;

    // If this is a strict mode function, verify that argument names
    // are not repeated, and it does not try to bind the words `eval`
    // or `arguments`.
    if (strict || node.body.body.length && isUseStrict(node.body.body[0])) {
      for (var i = node.id ? -1 : 0; i < node.params.length; ++i) {
        var id = i < 0 ? node.id : node.params[i];
        if (isStrictReservedWord(id.name) || isStrictBadIdWord(id.name))
          raise(id.start, "Defining '" + id.name + "' in strict mode");
        if (i >= 0) for (var j = 0; j < i; ++j) if (id.name === node.params[j].name)
          raise(id.start, "Argument name clash in strict mode");
      }
    }

    return finishNode(node, isStatement ? "FunctionDeclaration" : "FunctionExpression");
  }

  // Parses a comma-separated list of expressions, and returns them as
  // an array. `close` is the token type that ends the list, and
  // `allowEmpty` can be turned on to allow subsequent commas with
  // nothing in between them to be parsed as `null` (which is needed
  // for array literals).

  function parseExprList(close, allowTrailingComma, allowEmpty) {
    var elts = [], first = true;
    while (!eat(close)) {
      if (!first) {
        expect(_comma);
        if (allowTrailingComma && options.allowTrailingCommas && eat(close)) break;
      } else first = false;

      if (allowEmpty && tokType === _comma) elts.push(null);
      else elts.push(parseExpression(true));
    }
    return elts;
  }

  // Parse the next token as an identifier. If `liberal` is true (used
  // when parsing properties), it will also convert keywords into
  // identifiers.

  function parseIdent(liberal) {
    var node = startNode();
    node.name = tokType === _name ? tokVal : (liberal && !options.forbidReserved && tokType.keyword) || unexpected();
    tokRegexpAllowed = false;
    next();
    return finishNode(node, "Identifier");
  }

});

Smart Pascal FastCode inspiration

Remember the Fastcode project? The alternative memory management and memory operation routines which almost doubled the speed of Delphi 5-7 applications? And they did it by replacing the bog standard code from Borland with hand-optimized assembly language alternatives.

The opus magnum of Nerdvana

Well, today I wrote the Smart Pascal equivalent of the Fastcode project for Delphi. Or at least that’s what it reminded me of. So here is the Smart Pascal variations of the “Fastcode” like replacement modules 🙂

And yes, I did test these against 2-3 variations of the same procedures and these are as fast as I can make them without introducing lookup tables.

How fast are these routines? Well, the following code snippet was used for benchmarking:

for x:=1 to 1000000 do
begin
  mMemory:=AllocMem(1024);
  try
    WriteMem(mMemory,14,TDatatype.BytesToTypedArray
    ([10,20,30,40,50,60,70,80,90,100]));
    mMemory:=ReAllocMem(mMemory,512);
    Readmem(mMemory,14,10);
  finally
    Freemem(mMemory);
  end;
end;

The original code executes with time-code 551855 (which is approx 2.3 seconds) while this new variation executes with time-code 131816 which is a quite significant boost! But this is naturally because I have completely bypassed marshaled pointers which we are about to introduce in the next update of Smart Mobile Studio.

Pointers? Really?

As explained earlier, a marshaled pointer is ultimately an object which is managed, or takes care of something for you. In this case it assumes the role of a pointer, taking care of the reference to the memory segment (TMemoryHandle, as used above) and the offset into that segment.

The methods below does the exact same as those in the RTL, except they do not create a marshaled pointer at all. So this is for advanced Smart Mobile Studio developers only. If you hate low-level stuff and abhor getting your hands filthy from contact with JavaScript –then you probably dont want to mess with code like this in the first place.

But for the rest of us that like to squeeze every drop of performance out of both clients and server, this will give you something to play with. And it’s good practice for learning to optimize code which targets the most widely used virtual-machine in the world, namely the browser. A technology which eventually will take over the world and replace most of our native systems. We are a few years ahead of our time with SMS, but hey- world domination is hard work!

If you need more into on pointers under Smart Mobile Studio, I described that in detail here. You may also want to have a peek at this article and finally some inside info on the imminent update.

function  AllocMem(const Size:Integer):TMemoryHandle;
begin
  if Size>0 then
  Result:=new JUInt8ClampedArray(Size);
end;

procedure Freemem(var Memory:TMemoryHandle);
begin
  if (memory) then
  begin
    // decouple buffer from type
    // this does not release memory, but "hints" to the GC
    // to mark the segment for level 1 release classification
    JUInt8ClampedArray(Memory).buffer := NIL;
    Memory := null;
  end;
end;

function ReAllocMem(Memory:TMemoryHandle;
         Size:Integer):TMemoryHandle;
begin
  if (Memory) then
  begin
    if Size>0 then
    begin
      if Memory.length > Size then
      Memory:=JUInt8ClampedArray(memory).SubArray(0,Size-1);
      result:=new JUInt8ClampedArray(Size);
      JUInt8ClampedArray(result).Set(JUInt8ClampedArray(Memory),0);
    end;
  end else
  result:=AllocMem(Size);
end;

function WriteMem(const Memory:TMemoryHandle;
         const Offset:Integer;
         const Data:TMemoryHandle):Integer;
begin
  if (Memory) then
  begin
    if (Data) then
    begin
      if offset + data.length-1 > memory.length then
      result:=(offset + data.length-1) - memory.length else
      result:=data.length;

      if offset + Data.length > memory.length then
      JUInt8ClampedArray(Memory).Set(
      new JUInt8ClampedArray (JTypedArray(
      JUInt8ClampedArray(data).buffer.Slice(0,result-1))), offset) else
      JUInt8ClampedArray(Memory).Set(JTypedArray(data),offset);
    end;
  end;
end;

function ReadMem(const Memory:TMemoryHandle;
         const Offset:Integer;
         Size:Integer):TMemoryHandle;
begin
  if (Memory) then
  begin
    if Offset>=0 then
    begin
      if offset + Size-1 > Memory.length then
      dec(Size,(offset + Size-1)-Memory.length);
      result:=new JUInt8ClampedArray(JTypedArray(
      JUInt8ClampedArray(Memory).buffer.Slice(Offset,Offset + Size)));
    end;
  end;
end;

Next update, a closer look behind the scenes

With the next update right around the corner, what can you expect to find? Well, I am going to give you some insight and preview into the update right here. For some of you it will be like Christmas and new-years eve all rolled up into one. Especially custom control writers who need fine-grained accuracy for their HTML5 components. So let’s visit the secret lab of object pascal fundamentalists and have a peek 🙂

The secret lab that shall not be named

Functional handles

If you know your way around the VJL (visual JavaScript library) then you will notice a few minor changes here and there, especially that we now have more handle types. So far all browser specific types have been references by the THandle type, which is of type variant and thus capable of abstracting everything.

Starting with this update we begin the journey to a fully typed RTL; This means that the ultimate goal is to get rid of THandle all together and replace it with a typed reference to the HTML5 element itself. So instead of a “Handle:THandle” property in TW3EditBox you would actually get “Handle:JInputElement” which gives you direct access to the element from your pascal code.

But we are not there yet. So for now we have begun separating different handle types, which means that user-controls are no longer packing THandle as their handle-property, but rather TControlHandle.

Why did we do this? Well it’s simple. By separating handles into distinct types we can attach helper classes to them, simplifying how you work with and access handles. This allows us to replace one handle type at a time (without breaking backwards compatibility) and do so in a timely manner.

TControlHandle introduces the following helper methods:

  TControlHandleHelper = helper for TControlHandle
    function  Valid:Boolean;
    function  Ready:Boolean;
    function  Parent:TControlHandle;
    function  Root:TControlHandle;
    procedure ReadyExecute(OnReady:TProcedureRef);
    procedure ReadyExecuteAnimFrame(OnRead:TProcedureRef);
    function  Equals(const Source:TControlHandle):Boolean;
  end;

The ReadyExecute() method requires special attention. In short, whenever you create a visual control it can take a few milliseconds before it actually appears. This means that your code both inside the control and outside stand the risk of accessing a control which is not yet attached to the DOM (document object model).

ReadyExecute solves this by attaching a testing routine to this job, namely to continuously check if the handle is ready for use (read: injected into the DOM and is thus capable of being displayed). When the control is ready it will execute the anonymous procedure you provided.

This allows us to do things like this inside our constructor:

Constructor TMyListControl.InitializeControl;
Begin
  inherited;
  Handle.ReadyExecute( Procedure ()
    begin
      BrowserAPI.RequestAnimationFrame( procedure ()
      Begin
        ReSize;
        LayoutChildren;
      end;
    end);
end;

What the above code does, is to initialize the control as usual, but instruct the system to call-back immediately when the handle is inserted into the DOM. At which point we ask the browser to redraw AFTER we have resized and done layout of child elements.

This results in your control smoothly appearing inside it’s container. You don’t need to issue a resize or “nudge” to force a the layout manually, like some controls require right now. All of that is gone and we will revamp our entire control library with new and updated code.

We also have a method which includes the RequestAnimationFrame call automatically, cleverly named ReadyExecuteAnimFrame(). This is a handy method to make use of:

Constructor TMyListControl.InitializeControl;
Begin
  inherited;
  Handle.ReadyExecuteAnimFrame( Procedure ()
    begin
        ReSize;
        LayoutChildren;
    end);
end;

The Root Method is a great way to check if a control is actually attached to the DOM. If it’s attached to the DOM then the root element will be HTMLBodyElement. So this is a handy way of testing for this particular state.

Valid is as the name implies a method for validating that a TControlHandle is, well.. Valid! A valid reference means that it’s assigned a value and that it’s not null. Which would be the case for an invalid reference. It also makes your code look more human:

if Handle.Valid then
Begin
  if not Parent.Handle.Equals(w3form1.handle) then
  Raise Exception.Create("This control cannot be dropped directly on a form");
end;

The parent method will return the immediate parent, it will not traverse the DOM to locate the first owner, but rather just whatever element you have created an element under (form, panel, custom control).

The Ready method does the exact same thing as ReadyExecute, except that it simply performs the test. It doesnt create a time object which keeps checking for a state change, nor does it call back. This is an excellent way to find out of the current control is attached and ready to use.

Design considerations

I have written by font measuring before. I realize that to some people this may not seem very important, but you have to remember that HTML and JavaScript doesn’t have any functions for measuring font metrics (!). Thats right, these things doesn’t exist in the browser. So being able to measure the width and height of a piece of text, taking font-face and style into consideration — is no small feat. In fact you can’t even enumerate what fonts the browser supports, which is another aspect of accuracy we now provide as standard.

Starting with the next update, TW3MovableControl (base class) introduces these methods:

    function getFontInfo:TW3FontInfo;overload;
    function MeasureText(aContent:String):TW3TextMetric;overload;
    function MeasureTextFixed(aContent:String):TW3TextMetric;overload;

    class function MeasureTextSize(const aHandle:THandle;
          const aContent:String):TW3TextMetric;

    class function MeasureTextSizeF(const aHandle:THandle;
          const aWidth:Integer;const aContent:String):TW3TextMetric;

    class function getFontInfo(const aHandle:THandle):TW3FontInfo;overload;

Attributes

This is a slightly more complex feature which may not seem immediately valuable to some, but believe me, a lot of software could not be written without them.

A couple of years ago browsers began to support “user defined attributes”. In short it means that you can now attach your own attributes to HTML elements. The rules are simple: always pre-fix your attributes with “data-” and the browser wont ignore them. If you don’t follow that rule the browser just ignores them (or deletes them).

Let me explain just how important these are:
This update introduces the ability to apply effects with a single call to any custom-control. You can also chain effects together and they will execute in-turn, one after another. Cool right?

Absolutely, but how do you think this was done? The problem is this: Effect objects are created when you call them, but there has to be a list or queue type of mechanism which prevents them from starting all at once. There must be some way of telling effects waiting in escrow – to wait until the control is ready for more effects.
Since effect objects cannot be kept track of easily, not without an expensive list of stack table – the best way to inform the waiting effects that the object is busy, is to write the value to the element itself!

And this is exactly how we solves having multiple effects waiting their turn. When an effect is active, it writes a busy-flag to the HTML element. The other effects keep checking this flag and only execute when the object is not-busy.

This would not be possible without attribute access. Well, at least not without a rather expensive lookup table and cleanup procedure.

Other uses of attributes

Loads. You can now “tag” your own elements with whatever values you like. Perhaps you want to link a visual control to some data? Well, then you would define table and field as attributes. And when the data is ready, query the DOM through a selector to return those elements marked with your “tags”‘s. Voila you can now freely target and populate visual controls without even registering them. A sort of weak-binding type solution.

Visual effects

I mentioned above that the QTX effects library has been built into the RTL. And that is true. This makes it child’s play to make your UI designs become more responsive and alive by responding with effects to touches and callbacks. But don’t over-do it! To much can render your application useless, so use with care!

All TW3CustomControl decendants now supports the following effects:

1. fxFadeOut
2. fxFadeIn
3. fxWarpOut
4. fxWarpIn
5. fxZoomIn
6. fxZoomOut
7. fxScaleTo
8. fxMoveTo
9. fxMoveBy
10. fxMoveUp
11. fxMoveDown
12. fxSizeTo
13. fxScaleDown
14. fxScaleUp

These effects can be daisy-chained to execute after each other, like so:

  w3Button1
    .fxFadeIn(0.2)
    .FxMoveDown(1.2)
    .fxFadeOut(0.3);

Legacy

For all your FreePascal and Delphi coders we have a little treat, namely some classes which will simplify your HTML5 graphics programming life. Like most Delphi programmers you are probably used to good old TCanvas and TBitmap to do your work. Perhaps you want to port over some older Delphi code and cringe at using HTML5’s canvas to do the work?

Well we have re-implemented both TBitmap, TCanvas, TBrush and TPen for you. So go straight ahead and do that port, also be amazed at the wonderful speed enhancement HTML5 offers over native Delphi.

Streams, buffers and memory

I have covered these features before, you can read more about these here:

• Head to Head: Smart Mobile Studio vs. Embarcadero Delphi
• System.Interop, How to for Smart Mobile Studio

Dataset support

Yes. Finally. I know you guys have been asking about this since the beginning. Well starting with the next update we ship TW3CustomDataset and TW3Dataset, which is an in-memory dataset solution.

NodeJS builder will make the current architecture make more sense

It does not connect to anything and is more like TClientDataset in Delphi, but you can use it to cache records you want to send via DataSnap, or via RemObjects, and naturally it’s a perfect match for nodeJS services. It will all make more sense when we ship the nodeJS service designer (ops! Perhaps I should not have mentioned that part).

You will find that in system.dataset, which means you can also reference it from your nodeJS projects. Communication between server and client will be 100% compatible since dataset can be deployed on both sides.

Later I will introduce binary records (everything is now JSon) and in-memory storage which will be super-fast and super efficient. So we havent even begun scratching the surface of databases under HTML5!

And much, much more

These were just some of the features I am adding to SMS this time. We are a team of 4 developers so there will be plenty of bug-fixes and other features, both for the IDE, the compiler and the RTL.

I cant wait to get to work on the next update! There are so much cool stuff in the pipeline I hardly know where to begin!

Head to Head: Smart Mobile Studio vs. Embarcadero Delphi

I know I have been writing a lot about Smart Mobile Studio and memory access lately. But I’m sure you all understand that it’s just as vital in HTML5 applications as it is under native Delphi or C# applications.

V8 JIT powered JavaScript is awesome, even on embedded platforms

Having spent the better part of this weekend optimizing and re-factoring the codebase, I am now able to write 1.000.000 (one million) records to memory in 417 milliseconds. At first I did not believe that it could be possible, there had to be some error somewhere or perhaps the data was not actually written. But using hexdump and inspecting the buffer avails that yes – there is no error, it’s that bloody fast.

    mChunk:=0;
    mMake.Allocate(21000021);
    mStart:=Now;
    for x:=0 to 1000000 do
    begin
      mMake.Write(mChunk + 0,$AAAA0000);
      mMake.WriteFloat32(mChunk + 4,19.24);
      mMake.WriteFloat64(mChunk + 8,24.18);
      mMake.Write(mChunk + 16,false);
      mMake.write(mChunk + 17,$0000AAAA);
      inc(mChunk, 21);
    end;
    mStop:=Now;

This means that JavaScript, in particular Webkit V8 (JIT) code is now faster than native Delphi for standard memory operations (move, fill, read, write).

Graphics humiliation

Some five years back when I proposed on my earlier blog that a compiler producing JavaScript from Object Pascal could be constructed, people laughed. One of the biggest challenges when creating something new is that you face the general consensus of people stuck in the past, fixed to rigid ideas which may or may no longer apply to reality.

This was also the case with Delphi and our community back then. To the majority of programmers at that point in time, JavaScript was deemed a toy; it was to slow, unserious and unstable for anything real and tangible. A proposal that JavaScript could in fact be used as a universal platform for mobile and desktop applications was laughed out of the chat rooms.

Introduction of new ideas is often met by heavy resistance

Thankfully a genius french developer, Eric Grange, which is the maintainer of Delphi Web Script (DWS) came to my aid. What he did was create a HTML5 canvas demo which outperformed Delphi by magnitudes. First using FireFox which is as we all know slightly slower than Webkit, but later he executed the same code in Webkit’s V8, leaving native Delphi in the dust.

So the laughter I had faced was quickly silenced. I had been vindicated and Eric and myself went on to write what to day is Smart Mobile Studio. Eric adapted Delphi Web Script to serve as the compiler framework, adding a JavaScript code-generator which not only realized my ideas; It also went far beyond anything I had envisioned, with a VMT (virtual method table), class inheritance, interfaces, operator overloading, partial classes and the works.

The moral of the story here is that we were never out to humiliate Delphi. Quite the opposite, Smart Mobile Studio was created as a toolkit for Delphi developers. In fact it was created to help Embarcadero venture into new markets. I would never suspected to be treated like we have been by Embarcadero, which has blacklisted our entire team from public events, threatening to poll funding if we are allowed to speak (!)

But it has made one thing clear. I no longer care if Embarcadero is damaged by my products.  With the exception of two individuals working for EMB, they have not given me (or my team) the time of day. Which is strange, because our technology would revolutionize and to some degree re-vitalize the Delphi brand. Something it sorely needs.

Memory matters

Well, today I am pleased to announce that not only does Smart Mobile Studio humiliate Delphi in every graphics operation known to mankind, including 3D rotation of user-controls; Smart Mobile Studio is now faster in every aspect of memory manipulation. This includes:

• Writing un-typed array to memory
• Writing typed arrays to memory
• Moving data between two memory segments
• Moving data within the same memory segment
• Reversing the data inside a memory segment
• Fill a memory segment with char data
• Fill a memory segment with un-typed data
• Fill a memory segment with typed-data

The tests were all done on VMWare Fusion running on an 2.7 Ghz Intel Core i5 iMac with 8 GB memory (1600 Mhz DDR3).
1 Gigabyte of memory was allocated for the VMWare machine, which is running Windows 7 Ultimate Service Pack 2. The tests were performed using Chrome (Smart Mobile Studio ships with Chromium Embedded) and Chrome for OS X.

Webkit has excellent profiling capabilities, here showing profile for 1.000.000 writes

 

Inspection of memory and CPU timeline

The following Delphi versions were used in my tests:

• Delphi 7
• Delphi 2005
• Delphi 2006
• Delphi XE
• Delphi XE3
• Delphi XE6
• Delphi XE7

I also performed the tests using the latest FreePascal compiler, and I am happy to report that FreePascal is the only compiler capable of delivering roughly the same amount of power as V8 (the JavaScript runtime and JIT engine under webkit is called V8).

To be perfectly honest, I have begun to re-compile more and more Delphi projects using FreePascal and Lazarus these days. And we are moving Smart Mobile Studio to FreePascal as well. It gives us the benefit of compiling our product for OS X, Linux and Win32.

It also puts to rest any clause in the Embarcadero License which prevents users from producing executables from a Delphi application (yeah, it’s that screwed up!). So yes, a FreePascal powered version of Smart Mobile Studio will appear sooner or later.

Test yourself

The memory management unit, support for streams and marshaled pointers will be available in the next update of Smart Mobile Studio, so you can download a trial version if you dont own a license and run the tests yourself.

The full source-code will be posted here shortly, it’s identical to the SMS version in both payload and infrastructure.

System.Interop “how to” for Smart Mobile Studio

Smart Mobile Studio and the dialect “Smart Pascal” has roots in the Object Pascal community. The compiler and toolkit is written purely using FreePascal and Delphi, and naturally the run-time library for Smart Pascal is heavily influenced by both.

Background

What is less known is that Smart Mobile Studio in general, both the IDE and the RTL architecture, is equally inspired by the Mono project. The entire layout of the IDE is almost identical to Mono-Develop as it existed 4 years ago, while the RTL and application framework borrows from MonoTouch by exposing central constructs (like TApplication) purely through partial classes. These are language features which does not exist in FreeePascal or Embarcadero Delphi.

The unit Interop.pas which is a part of the system namespace (system.interop) is inspired by the mono .NET equivalent. Under Mono the various memory and type transformation classes are spread out over many different locations. Under Smart Mobile Studio we have decided to collect and implement these .NET features in a single unit. Namely system.interop (short for: System interoperability).

The Smart RTL is seriously fast

Memory management under HTML5

JavaScript does not expose any form of memory management. The closest thing to allocating memory under JavaScript is through JArrayBuffer and JTypedArray. JArrayBuffer is an un-typed memory segment without read or write access. The buffer can only be accessed by creating a descendant of JTypedArray and connecting that to JArrayBuffer. As such:

/* Allocate memory buffer, 1024 bytes */
var memBuffer = new ArrayBuffer(1024);

/* Allocate R/W access through a typed array */
var memAccess = new UInt8Array( memBuffer );

Needless to say, working with memory like this quickly becomes tiredsome, error prone and messy. Especially when coming from a FreePascal or Delphi background, where a simple call to AllocMem() is enough to reserve memory on the heap.

Memory Management under Smart Mobile Studio

The Interop unit introduces 3 different means of working directly with memory, there are:

• Streams
• Memory Buffers
• Marshaled classical memory access

Marshaled memory access

Due to the shortcomings of JavaScript the only way to expose and work directly with memory is through a technique called marshaling. It essentially means that you are abstracted from pure machine localized addresses (pointer types) and instead operate with reference objects (location objects).

Such a reference object contains two pieces of information:

• A reference (handle) to the associated JArrayBuffer
• An entrypoint (offset) into the arraybuffer

For instance, under classical object pascal the following code is quite common:

// pointer type containing the address
var
mBuffer: PByte;

// Allocate memory
mBuffer:=AllocMem(1024);
try
  // Work with memory here
finally
  // Release memory
  freemem(mBuffer);
end;

Using marshaled objects Smart Mobile Studio is capable of delivering the exact same:

// pointer type containing the address
var
mBuffer: TAddress;

// Allocate memory
mBuffer:=TMarshal.allocMem(1024);
try
  //work with memory here
finally
  // Release memory
  TMarshal.freemem(mBuffer);
end;

Marshaled references also have the benefit of helper functions. So advancing further into an allocated segment is done via the TAddress.Addr() function.

// pointer type containing the address
var
mBuffer: TAddress;

// Allocate memory
mBuffer:=TMarshal.allocMem(1024);
try
  mBuffer:=mBuffer.Addr(2); //Advance reference by 2 bytes
finally
  // Release memory
  TMarshal.freemem(mBuffer);
end;

Since JavaScript is garbage collected we can also approach memory offsets quick and dirty, like below; Fact is we really dont need to call FreeMem() because the garbage collector will come around and clean up after us later. But sloppy code has a proven negative impact on performance (memory fragmentation leads to sudden cpu spikes as the GC get’s busy), so I strongly advice you to stick with best practices object pascal.

var mBuffer: TAddress := TMarshal.allocMem(1024).addr(2);

Access to memory with such elegance is unheard of in the world of JavaScript. Marshaled reference objects can be found in many languages, but we are the first to introduce this for JavaScript. The speed benefit is likewise phenomenal compared to other solutions. In fact, our initial read/write tests outperformed all versions of native Embarcadero Delphi prior to Delphi XE.

Memory buffers

While marshaled references and memory allocations are essential for legacy compatability, porting code from freepascal or Delphi to HTML5 – being able to directly interact with a memory segment is vital for any modern platform.

As such, the Interop unit provides the TMemory class. Using the methods of this class you can allocate, release, move, copy, export, read and write directly to memory. The class implements caching for maximal speed enhancement. All intrinsic JavaScript datatypes are supported (Int16, Int32, Float32, Float64, String, Boolean).

Special attention should be made to the class-procedure Move() and Fill() which are equal to the classical FreePascal and Delphi variations. Unlike their classical counterparts these methods accepts TMemoryHandle rather than Pointer or marshaled TAddress objects.

Using Move with Marshaled Addresses

The TAddress class exposes a property named “Segment” of type TMemoryHandle. This can be used with Move() and Fill() as demonstrated below:

var
  src: TAddress;
  dst: TAddress;

TMemory.Move(
        src.segment,     // source memory
        src.entrypoint,  // source offset
        dst.segment,     // target memory
        dst.Entrypoint,  // target offset
        500              //bytes to move
        );

TMemory.Fill(
        src.segment,
        src.entryPoint,
        500,
        TDatatype.CharToByte("Z")
        );

It must be noted that the above code is just for explanation. Both Move() and FillChar() are implemented in the TMarshal class. The above demonstration is just to provide reference material on their use and how marshaled memory works.

Converting data

JavaScript is essentially type-less. The concept of “typed-arrays” was implemented a while back because they represent a significant speed boost. No matter if you are working on complex calculations or want to manipulate pixels on byte level, being able to work directly with memory is essential for inter-operability with other languages and file-formats.

Converting intrinsic (built in types) to binary can be very hard under JavaScript. As of writing, Smart Mobile Studio provides the most accurate and fastest means of doing this.

The class TDatatype provides methods for converting intrinsic datatypes to “array of bytes” and also from TByteArray back into intrinsic values. The class itself uses speed optimalization by pre-allocating conversion space when the unit is loaded into memory. This makes data conversion extremely fast, much faster than any other JavaScript solution out there.

Using TDatatype is very easy:

var
  x: Integer;
  mData: TByteArray;

// convert int32 value $1200 into 4 bytes
mData := TDatatype.Int32ToBytes($1200);

// write bytes to the console
for x:=0 to mData.length-1 do
writeln(mData[x]);

// Alter first byte just for fun
mData[0]:=$FF;

// Write modified integer value to the console
Writeln( TDataType.BytesToInt32(mData) );

The following methods are provided by TDatatype for conversion of datatypes:

  TDatatype = Class
  public
    class property  Bits:TBitAccess;
    class function  Int16ToBytes(Value:Integer):TByteArray;
    class function  Int32ToBytes(Value:Integer):TByteArray;
    class function  Float64ToBytes(Value:Float64):TByteArray;
    class function  Float32ToBytes(Value:Float32):TByteArray;
    class function  StringToBytes(Value:String):TByteArray;
    class function  BooleanToBytes(Value:Boolean):TByteArray;

    class function  BytesToInt16(const Data:TByteArray):Integer;
    class function  BytesToInt32(const Data:TByteArray):Integer;
    class function  BytesToFloat32(Const Data:TByteArray):Float;
    class function  BytesToFloat64(Const Data:TByteArray):Float;
    class function  BytesToString(const Data:TByteArray):String;
    class function  BytesToBoolean(const data:TByteArray):Boolean;

    class function  StrToBase64(Value:String):String;
    class function  Base64ToStr(Value:String):String;

    class function  ByteToChar(Const Value:Byte):String;
    class function  CharToByte(const Value:String):Byte;

    class function  StrToTypedArray(value:String):TJSByteClass;
    class function  TypedArrayToStr(const value:TJSByteClass):String;
  end;

Working with bits

The TDatatype class exposes a class-property named TBitAccess, which contains only class methods for manipulating bits. As of writing the following methods exists:

  TBitAccess = Class
    public
      class function  Get(Const index:Integer;Const Value:Byte):Boolean;
      class function  &Set(Const Index:Integer;Const Value:Byte;
                      const Data:Boolean):Byte;
      class function  Count(Const Value:Byte):Integer;
      class function  ToByteString(const Value:Byte):String;
      class function  FromByteString(const Value:String):Byte;
      class function  Dismantle(const Value:Byte):TByteArray;
      class function  Assemble(const Value:TByteArray):Byte;
  end;

Note: TMemory also exposes getBit and setBit methods for the whole allocated memory. This allows you to use a whole segment as a bit-buffer. This is often used by compression, sound generators and classes which communicates with hardware (see Smart Support for hardware platforms).

Working with Streams

While marshaled memory allocations and TMemory represents fast and direct access to raw data, RTL software typically base themselves on streams. In essence Streams and buffers represent the same thing, except that a stream maintains a cursor which is automatically updated as you advance through the content.

Smart Pascal implements the base-class “TStream” which is a purely abstract stream class, just like it exists under C#, FreePascal and Delphi. Deriving from this is TMemoryStream and TStringStream. We will no doubt add more streams to the mix at a later date, but due to the nature of HTML5 and the restrictions on JavaScript, file-streams serve little purpose at this point in time.

Using streams is done much like under FreePascal and Delphi:

var
  mStream: TMemoryStream;

mStream:=TMemoryStream.Create;
try
  // use stream here
finally
  mStream.free;
end;

Reading and writing to streams can either be done directly using TStream.Write() or TStream.Read(), both these methods accepts an array of byte (TByteArray). This makes streams slower to use than TMemory or TMarshal operations.

var
  mStream: TMemoryStream;
  mWriter: TStreamWriter;

mStream:=TMemoryStream.Create;
try
  mWriter:=mStream.createWriter;
  mWriter.WriteInt32($1200);
  mWriter.WriteString("This is how you write to a stream!");
  mWriter.WriteFloat64(139.68504);
finally
  mStream.free;
end;

Reading is done via TStreamReader, which you can either create manually or obtain via the TStream.CreateReader() method.

Speed comparison

JavaScript is seriously fast. Much faster than you would imagine, all things considering.

In the first test, which writes 100.000 records into a pre-allocated TMemory instance, we get the following results:

    mMake.Allocate(2100021);
    mStart:=Now;
    mChunk:=0;
    for x:=0 to 100000 do
    begin
      mMake.Write(mChunk + 0,$AAAA0000);
      mMake.WriteFloat32(mChunk + 4,19.24);
      mMake.WriteFloat64(mChunk + 8,24.18);
      mMake.Write(mChunk + 16,false);
      mMake.write(mChunk + 17,$0000AAAA);
      inc(mChunk, 21);
    end;
    mStop:=Now;

Time to completion: 0 seconds, 46 milliseconds

In the second test we omit pre-allocation, which will force TMemory to grow and re-allocate it’s cache quite often, depending on the size of the cache (4096 bytes in this case):

Time to completion: 0 seconds, 445 milliseconds

In the third test we do the exact same, but this time using streams and streamwriter. To fully understand what is happening here, for each write, consider the following:

1. StreamWriter calls TDatatype to convert the intrinsic value into a TByteArray
2. StreamWriter calls TStream to write the data
3. TStream maintains a TMemory class, and calls the rather slow Write() operation
4. TStream’s TMemory instance has no pre-allocation, and as such re-allocation will occur often

Time to completion: 0 seconds, 519 milliseconds

These numbers are almost ridicules. I thought something was wrong somewhere at first, and had to check that data was actually written and that it had written the data properly. But yes, the above numbers are correct and memory access under JavaScript when done right is beyond anything I have seen elsewhere.

As of writing my code is roughly two times faster than Perl and Python, it’s also faster than any native Embarcadero Delphi version except Delphi XE7. Only FreePascal and C# Mono compiled to machine-code using LLVM optimization is capable of performance like this. Which is to some degree embarrassing and humiliating for Delphi, a product costing 100 times more than Smart Mobile Studio.

On the positive side, it spells a bright future for Smart Mobile Studio, especially when interacting with nodeJS and hardware, which require powerful and fast code.

The Zen of Smart Pascal

The Zen of Smart Pascal

Something so elegant with memory management in place under Smart Mobile Studio. Moving pixels used to be so mind-numbingly slow compared to pointers. Now it’s a one-liner and faster than Embarcadero Delphi (only FreePascal is keeping pace).

The method ToBuffer() used to be over 200 lines long, as was LoadFromBuffer(). With the advent of TMemory, TMarshal and various low-level kick-ass to-the-bone methods now in place, such are things of the past. Now Smart Mobile Studio coders can enjoy the full power of modern OOP development under HTML5 using all the tools they are used to from FreePascal and Delphi.

function TW3Image.ToBuffer:TMemory;
begin
  if Ready then
  result:=TMemory.Create(toImageData.Handle.data);
end;

procedure TW3Image.LoadFromBuffer(const Memory:TMemory);
var
  mContext: TW3GraphicContext;
  mCanvas:  TW3Canvas;
  mData:  JImageData;
begin
  if Ready then
  begin
    mContext := TW3GraphicContext.Create(Null);
    try
      mContext.Allocate(PixelWidth,PixelHeight);
      mCanvas := TW3Canvas.Create(mContext);
      try
        (* Get pixel buffer *)
        mData:=mCanvas.Handle.getImageData(0,0,PixelWidth,PixelHeight);

        (* over-write pixel buffer with our data *)
        TMemory.Move(memory.Handle,0,Variant(mData.data),0,mData.data.length);

        (* put pixel buffer back *)
        mCanvas.Handle.putImageData(mData,0,0);

        (* And force a re-load via Base64 *)
        LoadFromUrl(mCanvas.ToDataURL("png"));
      finally
        mCanvas.Free;
      end;
    finally
      mContext.Free;
    end;
  end else
  Raise Exception.Create
  ('Loading image from memory failed, surface must be allocated first');
end;

Smart Mobile Studio and low level memory access!

For the past 7 .. 8 days I have spent most of my spare time working on two particular units: system.interop and smartcl.legacy. I don’t think I have ever spent so much time on a unit (at least not under SMS) for many years. Not because it’s super hard to code these, but because in order to arrive at the best solution – you often have to evolve your ideas through different approaches. You can read my previous post about this here.

But the journey has been worth it. Let me explain why

Memory allocations? But JavaScript doesn’t do that!

System.Interop, which takes its name from the .net framework, is all about code that facilitates interoperability; meaning classes and functions which helps you talk to other systems. Since “other systems” in many cases means native code, being able to work with untyped-memory (which JavaScript does support) and typed-array abstractions (called views, which conceptually functions like pointers) is essential.

I was able to port over both CRC and RLE compression
directly from my Delphi codebase without much change

I am happy to inform you that Smart Mobile Studio now supports a TMemory class, which encapsulates a fully untyped piece of memory. It extends ordinary and boring untyped-memory buffers with the ability to grow, shrink, scale and much more. All in all TMemory has around 50 methods for manipulating memory, so you are well catered for in that department.

So if you are half expecting some sloppy string buffer or something like that, you should be pleasantly surprised.

There is also a class called TDatatype which provides methods for converting intrinsic (built-in) datatypes to bytes, and bytes back to an intrinsic datatype. So you have handy single shot functions like Int32ToBytes, Float32ToBytes and naturally their counterparts (BytesToInt32, BytesToFloat32).

The problem is not the propblem, the problem is your attitude towards the problem.. savvy?

On top of this system we finally reach the pretty high level interface, namely streams. I write “finally” because this has been quite a chore. The central problem is not the browser, nor is it me. The central problem is that the wide majority of JavaScript developers blogging about their code doesn’t have a clue what they are talking about. A whole generation of programmers who never learnt about pointers at school or university (!) An intellectual emergency of epic proportions.

you can now use AllocMem() and the other classical memory management
functions just like you would under Freepascal and Delphi

Now dont get me wrong, I have seen some fantastic code out there and JavaScript is really the language which is seeing more growth than any of the others combined, but if you want to work with untyped memory and accessing them through “views” (read: typed pointers) then you preferably should have some experience with pointers first.

With my 20+ year background in programming I had no problems navigating the docs (once I found them), and although it took me a few days to sift through all the rubbish, separating the good bad, I feel we have truly power-house of a solution on our hands.

Speed

If you think JS is to slow for anything serious, think again! People were shocked when Delphi was humiliated by JavaScript graphics back in 2012. It turns out JavaScript is just faster than compiled Delphi (in this case Delphi 7 .. Delphi 2006). This was a wake-up call for many developers I think.

for x:=0 to 40000 do
begin
  mWriter.WriteInteger($AAAA0000);
  mWriter.WriteFloat64(19.24);
  mWriter.WriteFloat64(24.19);
  mWriter.WriteBoolean(false);
  mWriter.WriteInteger($0000AAAA);
end;

RECORDS  = 40.000
BYTES    = 29 bytes * RECORDS = 1,160.225 Megabytes
CACHE    = 4096 bytes
=======================
DURATION = 2.63 seconds

NOTE: The example above writes through a 4-layer architecture: StreamWriter to stream-object, stream-object to buffer via TDatatype (datatype conversion to bytes), buffer writes to memory. This is one of the more time consuming tests, since the dynamic nature of a stream forces the buffer to re-allocate and move the content frequently.

If we go below the stream architecture, straight to the TMemory class, well moving data around inside is blistering fast (the example above moves data through 4 different objects, which is the price to pay for a good framework). The reason low-level memory operations are fast is because un-typed buffers are allocated “as is” in memory; As a single block with a related pointer defined for the data. So forget everything you know about ordinary arrays under JavaScript;

PS: True pointers under JavaScript are called Typed-Arrays, and serve as a view or window into the allocated memory according to it’s type. So if you use an Int32 type you will naturally be able to access the memory as an array of integers. If you chose a byte pointer (UInt8Array) and connect that to the buffer – you can manipulate the buffer on byte level.

The hard part is creating an infrastructure around these methods, a bridge between intrinsic types and typed/un-typed memory. Which is exactly what System.Interop gives you.

Allocmem? No way!

My SmartCL.Legacy unit is all about helping you move your Delphi or FreePascal code and skill-set to Smart Pascal and HTML5. Since one of the major differences between FPC/Delphi and HTML5 is how graphics are drawn (the HTML5 canvas object bears little resemblence to good old TCanvas) I decided to create a full clone of our old belowed TCanvas. So if you have a ton of working Object Pascal code for drawing graphics around the place, you should now be able to re-cycle it under HTML5 with little or no change.

The same goes for TBitmap, a class that I must admit I have missed myself from time to time. It’s easy to create a graphics context and a TW3Canvas, but old habits die hard and TBitmap still has it’s uses. Even though we have been complaining about it’s limitations for decades 🙂

But what about classical pascal concepts like AllocMem(), FreeMem(), Move() and FillChar() I hear you say?

Well you may think I am joking but I implemented that as well! The hairs on your neck should stand up right about now. Yes you read right, you can now use AllocMem() and the other classical memory management functions just like you would under Freepascal and Delphi.

Dealing with pointers

If you know your JavaScript you may be thinking “But what pointers? JavaScript doesn’t have pointers!”. That is very true indeed. JavaScript doesnt have a type called pointer. So in order to provide such a type, we have to adopt some inspiration from C#: meaning that a pointer under Smart Mobile Studio is actually an object. This technique is called “marshaling”.

Here is how you allocate memory and play around with it:

procedure TForm1.testMemory;
var
  mData: TAddress;
begin
  mData:=TMarshal.Allocmem(1024);
  try
    //fill the buffer with zero
    TMarshal.FillChar(mData,1024,#0);

    //Fill 200 bytes, starting at offset 100, with the letter "A";
    TMarshal.FillChar(mData.Addr(100),200,"A");

    //move 200 bytes, starting at offset 200, to position 400
    TMarshal.Move(mData.Addr(200),mData.Addr(400),200);
  finally
    freemem(mData);
  end;
end;

Pretty cool right? And yes, AllocMem() really does allocate untyped memory. TAddress is just a very, very thin piece of code which serves as a “reference point” into the allocated untyped memory.

The result? Well, I was able to port over both CRC and RLE compression directly from my Delphi codebase without much change.

Marshaling

like mentioned, keeping references to memory through handles and offsets is called marshaling. It’s more or less the same technique as used by CLR under .NET, and before that Java and Visual Basic (and many more languages as well). So these languages have gotten around their initial lack of memory access without to much penalty. I mean – games like MineCraft is written in 100% Java and runs just as fast as native C++ despite the lack of pointers. As does the .NET framework (note: the “lack” of pointers is not really true for the .NET framework, but pointer operations are regarded as “unsafe”).

Marshaling simply means that TAddress contains a reference to the un-typed memory, and also an offset into this memory buffer (see TAddress.Entrypoint property). The “real” memory is only accessed through the operations which act on the memory, and only then through typed-views which ensure safety (sigh).

Let’s take a peek under the hood of TAddress

  TAddress  = Class
  private
    FOffset:    Integer;
    FBuffer:    THandle;
  protected
    function    _getSegRef:THandle;virtual;
  public
    Property    &Type:JUInt8ClampedArray
                read ( new JUInt8ClampedArray(JTypedArray(_getSegRef)) );
    Property    Entrypoint:Integer read FOffset;
    Property    Segment:THandle read FBuffer;
    function    Addr(const Index:Integer):TAddress;
    function    Reflect:TMemory;
    Constructor Create(const aSegment:THandle;
                const aEntrypoint:Integer);virtual;
    Destructor  Destroy;Override;
  end;

//############################################################################
// TAddress
//############################################################################

Constructor TAddress.Create(const aSegment:THandle;
            const aEntrypoint:Integer);
begin
  inherited Create;
  if (aSegment) then
  FBuffer:=aSegment else
  Raise EAddress.Create('Failed to derive address, invalid segment error');

  if aEntryPoint>=0 then
  FOffset:=aEntryPoint else
  Raise EAddress.Create('Failed to derive address, invalid entrypoint error');
end;

Destructor TAddress.Destroy;
begin
  FBuffer:=NIL;
  FOffset:=0;
  inherited;
end;

function TAddress._getSegRef:THandle;
begin
  result:=JTypedArray(FBuffer).buffer;
end;

function TAddress.Addr(const Index:Integer):TAddress;
var
  mTarget:  Integer;
begin
  if Index>=0 then
  Begin
    mTarget:=FOffset + Index;
    if (mTarget>=0) and (mTarget < JUint8ClampedArray(FBuffer).byteLength) then
    result:=TAddress.Create(FBuffer,mTarget) else
    raise EAddress.Create
    ('Failed to derive address, entrypoint exceeds segment bounds error');
  end else
  Raise EAddress.Create
  ('Failed to derive address, invalid entrypoint error');
end;

function TAddress.Reflect:TMemory;
begin
  if (FBuffer) then
  result:=TMemory.Create(FBuffer.buffer) else
  Raise EAddress.Create('Failed to reflect memory, null reference error');
end;

As you can see from the code above, TAddress is extremely efficient and thin. It only manages the entrypoint into the associated memory-segment, the rest is very simple but effective maintenance code.

Let’s take a closer look under the hood of TMarshal while we are at it:

//############################################################################
// TMarshal
//############################################################################

class procedure TMarshal.FillChar(const Target:TAddress;
      const Size:Integer;
      const Value:Byte);
var
  mSegment: JUint8ClampedArray;
  mIndex:   Integer;
Begin
  if Target<>NIl then
  begin
    mSegment:=JUint8ClampedArray( Target.Segment );
    if mSegment<>NIL then
    Begin
      mIndex:=Target.Entrypoint;
      TMemory.Fill(Target.Segment,mIndex,Size,Value);
    end;
  end;
end;

class Procedure TMarshal.FillChar(const Target:TAddress;
      const Size:Integer;
      const Value:String);
var
  mSegment: JUint8ClampedArray;
  mByte:    Byte;
Begin
  if Target<>NIl then
  begin
    if Value.length>0 then
    begin
      mByte:=TDataType.CharToByte(Value);
      mSegment:=JUint8ClampedArray( Target.Segment );
      if mSegment<>NIL then
      TMemory.Fill(Target.Segment,Target.Entrypoint,Size, mByte);
    end;
  end;
end;

class procedure TMarshal.Move(const Source:TAddress;
          const Target:TAddress;const Size:Integer);
Begin
  if Source<>NIL then
  Begin
    if Target<>NIl then
    begin
      if Size>0 then
      TMemory.Move(Source.segment,Source.Entrypoint,
      target.segment,target.entrypoint,Size);
    end;
  end;
end;

class procedure TMarshal.ReAllocmem(var Segment:TAddress;
                const Size:Integer);
var
  mTemp:  TAddress;
  mSize:  Integer;
begin
  if segment<>NIL then
  begin
    mSize:=JUint8ClampedArray(segment.Segment).length;
    mTemp:=AllocMem(Size);
    case (Size>mSize) of
    true:   move(segment,mtemp,mSize);
    false:  move(segment,mTemp,Size);
    end;

    //Ensure reference is picked up by garbage collector
    SegMent.free;
    Segment:=NIL;
    Segment:=mTemp;
  end else
  SegMent:=AllocMem(Size);
end;

class function TMarshal.AllocMem(Const Size:Integer):TAddress;
var
  mBuffer:  JArrayBuffer;
  mArray:   JUint8ClampedArray;
begin
  result:=NIL;
  if Size>0 then
  Begin
    mBuffer:=new JArrayBuffer(Size);
    asm
    @mArray = new Uint8ClampedArray(@mBuffer);
    end;
    result:=TAddress.Create(mArray,0);
  end;
end;

class procedure TMarshal.FreeMem(Const Segment:TAddress);
begin
  if Segment<>NIL then
  Segment.free;
end;

You could be excused to think that this can hardly provide such advanced features, and you are right. The infrastructure which does the actual moving of data or population of data is inside the TMemory class. Which is far to big to post here.

Well, I hope I have wetted your appetite for cutting edge HTML5 development with Smart Mobile Studio!

Enjoy!