Getting data into your Smart Mobile apps

The new reality is all about small, powerful mobile devices. The 80’s was all about home computing, the 90’s about creativity and the 2k’s about connectivity. Well we have mastered all of those aspects of computing, and now it’s the age of mobility! Not just for devices you put in your pocket, but for the data that comes with it.

Getting data

No matter if you are writing your next iPhone or Android application with Delphi + FMX or Smart Mobile Studio + Phonegap, your data has to come from somewhere. Unless it’s a very small, very limited notepad application where your customer is expected to populate it from top to bottom; but that would be a very strange app in our day and age.

Getting data from your company servers or perhaps the latest public service is not always easy. But with the advent of WebSockets this has pretty much changed – if you have paid attention to the web technology that is.

So what are WebSockets? In short it’s just like normal networking sockets, but with a few limitations. You are allowed to connect, send ordinary text and also binary data. The latter is in the form of blobs or untyped array buffers. Dont worry, we have taken care for all that for you, so sit back and relax and enjoy the code.

Also websockets are automatically created for long sessions, so the underlying architecture will try to keep the connection alive. This is very important since we dont want to re-connect over and over again just to send small packets. So it’s more economic to keep a single connection alive that both server and client read and write to.

Creating a server

First you need to download the WebSocket server (based on Indy which is installed with Delphi). This can be done from this website. If you havent written a server before, don’t worry – it’s actually quite fun! Especially with WebSocket since essentially what you do is read commands, fetch data, send a response – and then wait for another command again. But you should get a book on the subject if you are a complete newbie, or get acquainted with the concept by trying Indy from Delphi first.

Server design is also fun because you get to define the protocol your mobile devices should use! You can go for a simple text-based protocol, you can opt for superobject (JSON objects) based packets, XML or just invent something completely new. You may also want to stick to standards so that people can buy access to your services? Quite a few people make a living creating just web-services. It’s a rather lucrative market to be honest.

Note: I urge you to create your first server as a normal GUI program, and only later when everything is working 100% isolate it as a Windows Service or Linux Daemon. There are bound to be errors (network coding is no different from other types of coding) and it’s easier to debug and display those errors with a GUI.

Creating the client

Your apps liven up with data from a server

This is the fun part! Once you have some rudimentary server up and running, be it an echo service or a serious power-house of a database service, Smart Mobile Studio will help you deal with it.

Working with web-sockets is really not that different from using the ordinary Indy components. You call commands and handle events, like OnMessage which signals that a text has been received from the server. Remember that you can send and deal with binary data through Base64 encoding. JSON is perfect for stringifying complex datatypes – so everything is really for the picking here.

Below is a websocket class ready to be played with. Please note that this uses the new RTL classes, which means you have to be a part of the beta team to compile it. Although it will compile if you remove all the write() method except the string based one.

Either way, it demonstrates how easy it is to transport data between your company server – and your Smart Mobile Studio application. Websockets are super awesome, easy to use, they work on all platforms (and all popular browsers), on mobile devices just as desktop — so it’s the easiest route to get your data from A to B.

  TWebSocketHandle  = THandle;

  TWebSocketState = (stError,stIdle, stConnecting, stConnected, stClosing, stClosed);

  TWebSocketOpenEvent     = procedure (sender:TWebSocket);
  TWebSocketCloseEvent    = procedure (Sender:TWebSocket);
  TWebSocketErrorEvent    = procedure (Sender:TWebSocket);
  TWebSocketMessageEvent  = Procedure (Sender:TWebSocket;Value:String);

  EWebSocket = Class(EW3Exception);
  TWebSocket = Class(TObject)
  private
    FHandle:    TWebSocketHandle;
    FOnOpen:    TWebSocketOpenEvent;
    FOnClose:   TWebSocketCloseEvent;
    FOnMessage: TWebSocketMessageEvent;
    FOnError:   TWebSocketErrorEvent;
  public
    Property    OnOpen:TWebSocketOpenEvent read FOnOpen write FOnOpen;
    Property    OnClosed:TWebSocketCloseEvent read FOnClose write FOnClose;
    Property    OnMessage:TWebSocketMessageEvent
                read FOnmessage write FOnmessage;
    Property    OnError:TWebSocketErrorEvent
                read FOnError write FOnError;

    function    SocketState:TWebSocketState;
    Function    Connected:Boolean;
    function    URL:String;
    function    Protocol:String;

    Procedure   Connect(URL:String;Protocols:Array of String);
    Procedure   Write(value:String);overload;
    procedure   Write(Value:TMemoryHandle);Overload;
    Procedure   Write(Value:TStream);overload;
    Procedure   Write(Const Data:TBinaryData);overload;

    procedure   Disconnect;
    Destructor  Destroy;Override;
  end;

uses  W3C.DOM,
      W3C.TypedArray,
      W3C.WebSocket;

//############################################################################
// TWebSocket
//############################################################################

Destructor TWebSocket.Destroy;
Begin
  if (FHandle) then
  Disconnect;
  inherited;
end;

function TWebSocket.Protocol:String;
begin
  if (FHandle) then
  result:=JWebSocket(FHandle).protocol;
end;

function  TWebSocket.URL:String;
begin
  if (FHandle) then
  result:=JWebSocket(FHandle).url;
end;

function TWebSocket.SocketState:TWebSocketState;
	const
		CONNECTING: Integer = 0;
		OPEN: Integer = 1;
		CLOSING: Integer = 2;
		CLOSED: Integer = 3;
begin
  if (FHandle) then
  begin
    case JWebSocket(FHandle).readyState of
    CONNECTING: result:=stConnecting;
    OPEN:       result:=stConnected;
    CLOSING:    result:=stClosing;
    CLOSED:     result:=stClosed;
    else        result:=stError;
    end;
  end else
  result:=stIdle;
end;

Function TWebSocket.Connected:Boolean;
begin
  result:=not (SocketState in [stIdle,stClosed,stError]);
end;

Procedure TWebSocket.Connect(URL:String;Protocols:Array of String);
begin
  (* disconnect socket if already connected *)
  if connected then
  disconnect;

  (* Allocate new socket *)
  try
    asm
    (@self.FHandle) = new WebSocket(@url,@protocols);
    end;

    JWebSocket(FHandle).onclose:=Procedure ()
      begin
        if assigned(FOnClose) then
        FOnClose(self);
      end;

    JWebSocket(FHandle).onopen:=Procedure ()
      Begin
        if assigned(FOnopen) then
        FOnOpen(self);
      end;

    JWebSocket(FHandle).onmessage := procedure ()
    var
      event:  Variant;
    begin
      asm
        @event = event;
      end;
      if assigned(FOnMessage) then
      FOnMessage(self,String(event.data));
    end;

    JWebSocket(FHandle).onerror := procedure ()
    begin
      if assigned(FOnError) then
      FOnError(self);
    end;

  except
    on e: exception do
    Raise EWebSocket.CreateFmt
    ('Connect failed, system thew exception %s [%s]',[e.classname,e.message]);
  end;

end;

procedure TWebSocket.Disconnect;
begin
  if Connected then
  begin
    try
      try
        JWebSocket(FHandle).close();
      except
        on e: exception do;
      end;
    finally
      FHandle:=NULL;
    end;
  end;
end;

Procedure TWebSocket.Write(value:String);
begin
  JWebSocket(FHandle).send(value);
end;

procedure TWebSocket.Write(Value:TMemoryHandle);
begin
  JWebSocket(Fhandle).send(JArrayBufferView(Value));
end;

Procedure TWebSocket.Write(Value:TStream);
var
  mRaw: TMemoryHandle;
begin
  if Value<>NIL then
  begin
    if Value.Size>0 then
    Begin
      Value.Position:=0;
      var mBytes:=Value.Read(Value.Size);
      mRaw:=TDataType.BytesToTypedArray(mBytes);
      Write(mRaw);
    end;
  end;
end;

Procedure TWebSocket.Write(Const Data:TBinaryData);
begin
  if Data<>NIL then
  Begin
    if Data.Size>0 then
    Write(Data.ToTypedArray);
  end;
end;

SQLite + Smart Mobile Studio = True

In case you missed the great moment, here is a link to my previous post on the subject.

In short: Someone has ported the engine SQLite database engine to JavaScript. This means that you can now create, query, save, load and otherwise do whatever you wish without relying on the browser to provide you with a native DB API. This means that if you desire SQL database support in your mobile applications then Smart Mobile Studio now provides that out of the box!

I must mention though that we also provide our own lightweight TW3Dataset class, which for known table structures is much more efficient and faster to work with. But TW3Dataset does not have SQL support and searching for records must be done “the old way”. The SQLite engine applies internal indexes to make searching for data fast and simple, and you get to use SQL which can simplify complex expressions.

Libraries

The SQLite implementation is pure JavaScript and will be stored in the libraries folder. When you include the sqlite unit in your projects, this library is automatically included – and the file copied to your /res/ folder.

To initialize the SQLite driver you simply do like this:

procedure TForm1.InitializeForm;
begin
  inherited;
  w3Button1.enabled:=False;
  w3_RequestAnimationFrame( procedure ()
    begin
      SyncDBStart;
    end);
end;

Procedure TForm1.SyncDBStart;
begin
  if CheckDBEngine then
  begin
    FDB:=TSQLiteDatabase.Create;
    w3button1.enabled:=True;
  end else
  w3_RequestAnimationFrame(SyncDBStart);
end;

As you can see from this code, I disable the GUI while the database engine is loading. I use an async-loop to check if the driver has initialized, and only when true do I re-enable the GUI and create an instance of the database.

You may want to modify your application unit and make sure the driver is loaded before you show a form, or perhaps show a fancy CSS3 dialog with “Loading, please wait” until all your resources are in place? The new system.fileutils.pas unit should give you some inspiration there 😉

Excellent performance and great storage use

Well, here is a picture of the testbed app. Notice the bytesize of a SQLite database with 10.000 inserted records (!)
This means that you can easily stuff thousands of records into a database stored in LocalStorage (max 5 megabytes)! Perfect for lazy updates where you upload the data to your server when the device is connected to the internet.

Enjoy!