Home > Delphi, JavaScript, Object Pascal, Smart Mobile Studio > Smart Pascal Database, finally

Smart Pascal Database, finally

Work is progressing nicely on the Smart Mobile Studio update. The whole team is busy adding new features, fixing bug reports and testing code. It’s a wonderful sight to see more and more of our goals get into the product. We still have a long way to go with many years of exciting development ahead of us, but the foundation is pretty much rock solid for what has become the “Delphi” of our age.

Working hard on the update

Working hard on the update

Binary at last

Those of you that follow my blog know that real data, binary data, is a touchy subject for JavaScript programmers. This has motivated me to work even harder to bypass and overcome the challenges the platform represents. And needless to say, we have not just overcome the barriers, but we have managed to do so without turning to hacks or parlor tricks involving strings.

We are in fact the only HTML5 compiler suite to offer a 1:1 stream system 100% compatible with native languages. In fact, we even modeled the stream classes themselves after FreePascal and Delphi. This means that not only do they work as you expect, they also are exactly what you expect. Right down to the last method.

This has changed Smart Mobile Studio for my part. I loved working with SMS since the day it was born, but with the advent of memory allocations, marshaled pointers, streams and memory IO which gives native object pascal a run for it’s money, Smart Mobile Studio enters a new era.

More memory tools

The memory system in FreePascal and Delphi is rich. You have out of the box support for Allocmem and all the low-level methods that have been there since the last days of Turbo Pascal, followed by streams of various types, reader and writer isolation and a fast binary persistent framework: TFiler.

Well, we decided to do some changes to this old recipe. It’s not every day you get to write an object pascal RTL from scratch and define a language and dialect, so trying to get this right was tricky. You want to be backwards compatible, but not so much as to hinder future development and changes.

As of writing the code has been divided into the following sections

  • Unmanaged memory
  • Managed memory
  • Streams
  • Binary data
  • Datatype conversion

Unmanaged memory

As the name implies, this represents classes and methods for allocating memory which are un-managed by Smart Pascal classes. JavaScript has built-in garbage collection so we are not talking about that type of management, but rather management which comes into play when you use the same buffer from multiple places. Should the segment be cloned? Should it be split? This is what management is all about. TUnManaged and TMarshal are objects covering these topics.

Managed memory

This includes marshaled pointers, which takes care of references to memory segments and offset’s into those segments. This emulates the mechanics of a typed-pointer under FreePascal or Delphi. Associated with these are classes and methods allowing you to allocate, release, scale and move data between segments.

Streams

As of writing Smart Mobile Studio supports TStream, TMemoryStream, TStringStream, TReader and TWriters. More streams will be added, including ZLib compression and probably an extension to attach encryption codec’s.

Binary Data

The ability to convert intrinsic datatypes into bytes and bytes back to intrinsic datatypes. Also directly to un-typed memory segments. This is covered by TDatatype and various other classes. Also the core class, TBinaryData, which inherits from TAllocation provides a plethora of methods for altering memory segments, all the way down to bit operations.

Dataset and databases

Dataset + NodeJS = true

Dataset + NodeJS = true

This is the big one that many people are waiting for. Well, the update will not provide wizards and 1-2-3 connection steps for popular databases. That requires a bit more infrastructure (but we are working on such wizards, so dont worry, it’s on its way). But this is the point in time where we introduce TW3Dataset, which is an in-memory database table.

I have just added the methods required to store directly to binary format (streams). It supports JSON format already, so now we have a solution for both JS friendly code and native systems.

The thing about the dataset is that it’s tailored after Mono/C#’s dataset. Meaning that it’s ultra-light, fast, simple to use and easy to deploy. Perfect for sending data between clients and servers, not to mention in-memory caching of information. If you are coding a web-shop you would use a TW3Dataset in memory to keep track of products ordered, and then transport the entire dataset to your nodeJS or RemObjects SDK service when the customer clicks “Order”.

TW3Dataset is also written in Smart Mobile Studio. It’s not a wrapper around WebSQL or IndexDB which are severely limited. Want to persist it in the browser? No problem. Just save the dataset into LocalStorage and re-load it the next time a customer visits your website.

So now you can do cool stuff like:

uses System.Types, System.TypeConv, System.Memory, System.Streams, System.Dataset;

Procedure TApplication.InitializeObject;
begin
  inherited;
  //Create our application global dataset
  FAppTable:=TW3Dataset.Create;

  //Do we have data in browser storage? If so, load
  if FStorage.keyExists("appdata") then
  FAppTable.LoadFromStream(Storage.KeyToStream("AppData")) else
  begin
    //First visit or storage deleted, re-create table
    FApptable.FieldDefs.Add("id",ftAutoInc);
    FAppTable.FieldDefs.Add("session_key",ftGUID);
    FAppTable.fieldDefs.Add("customer_id",ftInteger);
    FAppTable.fieldDefs.add("userid",ftString);
    FAppTable.FieldDefs.add("last_login",ftDateTime);
    FAppTable.FieldDefs.add("shopping_basket",ftBlob);
    FAppTable.CreateDataset;
  end;
end;

I took the liberty of adding auto-generated fields, such as ftAutoInc and ftGUID. A GUID field is excellent for HTML5 development, because dictionaries in the world of JavaScript are name/value pairs. So GUIDs ensure entries have unique identifiers for serialization.

Another cool factor which is now possible to achieve, is ZLIB compression and power encryption (like RC4, Blowfish etc). As you know there is a limit to how much data you are allowed to store via the browser. The capacity varies from browser to browser but is typically in the 5-10 megabyte range. Thankfully our TW3Dataset is as lightweight as possible and will be able to pack a lot of data into 10 megabytes is structured properly.

It’s important to underline that this limitation only exists when you embed Smart Mobile applications into your webpage (like you would embed Flash applications earlier). The moment you compile your Smart Pascal application with Cordova / Phonegap, the limitation goes away and you have the same storage rights as other native applications.

But using Smart Mobile Studio as a complete replacement for Adobe Flash, or even better – write your entire website in Smart Mobile Studio in order to de-couple the presentation from the data layer, is now easier than ever before. Why spend hundreds of dollars on buying sliding banners and other website bling, when you can code it yourself in a couple of hours in Smart?

Kick ass compression

ZLIB compression is going to help with many aspects of web application development. For instance it will boost your ability for record caching; sending records in bulk back to the company server as opposed to calling the server once every time a customer adds a product to his basket.

In our model, the customer selects their products and these are stored in a TW3Dataset. Only when the customer is ready to use his or her’s credit-card and click the “Pay now” button should you transport the full dataset to the company servers for processing. If you are worried about safety, this is actually more safe than the majority of existing solutions. You would naturally run the page with SSL, disable source-code access etc. And by applying encryption and compression to your data, it’s going to be very hard for people to make use if it.

But what should you run server-side? Well, this is where Smart Mobile Studio’s support for NodeJS comes in. Because you write the server as well in Smart Pascal. NodeJS has excellent support for all types of databases (Oracle, MSSQL, MySQL, MariaDB and a bunch of others) so connecting to your database is easy enough.

All you have to write is the REST API you want to expose to your mobile or HTML5 web-application. With that in place you have effectively solves the entire IT infrastructure from A to Z with a single technology: Object Pascal.

Well — back to the code!

Advertisements
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: