Home > JavaScript, Object Pascal, Smart Mobile Studio > Binary data, taking JavaScript to the next step with Smart Mobile Studio

Binary data, taking JavaScript to the next step with Smart Mobile Studio

February 27, 2015 Leave a comment Go to comments
Buffers, stuffers and coffers

Buffers, stuffers and coffers

I guess the cat is out of the bag, an update to Smart Mobile Studio is immanent. To avoid people making stuff up based on rumors over at delphideveloper, I figured I would explain about the binary buffers and streams which will be introduced shortly. That way people can stop making up stuff and focus on what is really in the pipeline.

As you all probably know, JavaScript is not “native” friendly. Concepts like bytes, bits, memory access and even raw file access is utterly alien to JavaScript (in normal browser mode, nodeJS or custom IE scripts is another thing). If you want to work with data or chewing files in some form, you better be prepared for some off-the-wall string decoding.

In short: This way of working has bugged me so much that I decided to fix this once and for all, and in the next update of SMS you will be pleased to find TW3MemoryBuffer with accompanied streams.

Note: W3Buffers.pas is already in smart mobile studio, but that only provides the bare-bones and no transcoding or “byte” datatype. But SMS already has good support for the native JS framework.

The memory buffer is ultimately a wrapper around a typed array, in this case a UInt8Array (array of bytes). To make it as speedy as possible I have littered every move and copy operation with loop-expansion, handling 8-by-8 items per loop as opposed to a meager for/next solution. This makes moving data around as fast as it get’s under JS without resorting to C#/C++ extensions. We want our code to be pure, 100% HTML5, browser compliant JavaScript.

Also it supports grow and shrink methods that retain data (this is not supported by JS out of the box, it’s implemented by my classes), with all the bells and whistles you could want for: move(), copy(), clone(), read(), move data between buffers, copy from pixel-buffers .. the works!

If you are pondering what loop expansion is, and why it makes things faster, here is an example:

function TW3MemoryBuffer.ToRaw:THandle;
var
  x:  Integer;
  mLen: Integer;
  mLongs: Integer;
begin
  result:=null;
  mLen:=getSize;
  if mLen>0 then
  begin
    asm
      @result = new Uint8Array(@mLen);
    end;
    if (result) then
    begin
      x:=0;
      mLongs:=mLen shr 3;
      while mLongs>0 do
      begin
        JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
        JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
        JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
        JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
        JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
        JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
        JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
        JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
        dec(mLongs);
      end;

      case mLen mod 8 of
      1:  begin
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
          end;
      2:  begin
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
          end;
      3:  begin
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
          end;
      4:  begin
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
          end;
      5:  begin
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
          end;
      6:  begin
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
          end;
      7:  begin
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
            JUInt8Array(result)[x]:=JUInt8Array(FHandle)[x];inc(x);
          end;
      end;

    end;
  end;
end;

As you can see from the code above, we process 8 bytes per loop and finish off the reminder with a fast case-switch. This makes moving data very fast. We could also expand it further by processing 16 items per loop — but odds are such a high volume would actually degrade the speed on smaller data-chunks, so 8 is a good number for less than 2MB buffers.

Positivities

The benefit of all this is that you can now load in a binary file, read bytes, integers, booleans, floats or whatnot – just like you would under Delphi. It also means that input data is no longer limited to JSON format, so if you have a custom database engine which works with streams – chances are it will compile and run under Smart Pascal with little modification.

It also gives us a uniform way of transporting data between objects. As you probably know components like TW3Image, which is a container host for a pixel-buffer, supports “fromDataUrl”, which allows it to process and display graphics moved from TW3Canvas (toDataUrl method). Well TW3Image has been given a host of new properties and methods, including importing and exporting content to both buffer and stream (as well as a few others).

The same will go for TW3Dataset which is introduced shortly. A class which represents an in-memory dataset you can use when talking to DataSnap, REST services or Remobjects servers.

I hope that clears up any intrigue, rumor etc.

As for the rest of the update — well, it’s going to be pretty awesome! The above is just one single point on a very long list of extension and fixes! You will never look at HTML5 control writing quite the same again, so hang on tight for more news in due time.

And yes, the QTX Library is being merged into the RTL — which means chainable effects on every visible element is now possible. Imagine how cool your custom controls will become, when you can slide, fade and move things about in response to user-input?

Well, stay cool – it’s coming along nicely!

Advertisements
  1. abouchez
    March 3, 2015 at 7:35 am

    Why are you working at the byte level, for the copy?
    You can map the same buffer with a JUint32Array, then copy with 4 bytes in row.
    If you take care of the data alignment, it should be faster.
    I guess your buffer should not only be mapped as JUint8Array, but also JFloat32Array and JFloat64Array, and, with proper alignment, you have very fast access to the raw data.

    • Jon Lennart Aasenden
      March 3, 2015 at 8:15 pm

      I know. One step at the time 🙂 Changing datatype is the second optimization im doing, that and allocation stride-alignment.

      So the following switches are present, which users can set depending on target platform (older browsers should fall-back to byte operations without a view):

      {$DEFINE DATASIZE_FLOAT_LARGE}
      {$DEFINE ALLOCATE_VIEW}
      {$DEFINE USE_CACHE}
      {$DEFINE USE_BYTE_RANGE_TEST}

      Using views you can move 8 bytes per move (Float64). I have settled on ordinary longwords though, moving blocks of 16 bytes per loop with a trail-select-case from 0..15.

      Stride-align is used on buffer-allocation so all buffers are rounded up to nearest boundaries of 16 (stride must match loop expansion naturally).

      Also, perhaps more importantly, caching! The buffer now has an allocation cache of at least 1024 bytes. This makes buffer growth much faster since a re-size is only an integer adjustment.
      Only when the cache is depleted is the true buffer grown and re-populated.

  1. No trackbacks yet.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: