Home > Delphi, JavaScript, nodeJS, Object Pascal, OP4JS, Smart Mobile Studio > Smart Mobile, we want your ideas!

Smart Mobile, we want your ideas!

December 26, 2015 Leave a comment Go to comments

Smart Mobile Studio has so much potential and can go so many ways, so much in fact, that we want to hear your ideas on some important subjects. As a company we want to deliver what our customers need, rather than what we feel is appropriate.

So I will be airing various topics in the weeks to come, hoping that as many of you take the time to think, ponder and give us some ideas. If you have your own framework and feel it would be perfect for global adoption, then send it to our main website (www.smartmobilestudio.com) and we will have a look at it.

Database

This has been a hot potato for a while now. It’s been a tough nut to crack because JavaScript really is utterly data-agnostic. There is no default framework to adhere to in the browser, and as such – no real object model to implement.

To make things even worse, browsers ship with 3 different DB engines (WebSQL was by far the best, but it’s now marked as obsolete), all of them so different that orchestrating one common framework to encapsulate them all has turned out futile.

But all is not lost, we still have the coolest OOP JS compiler on the market and with a solid Delphi background, we can easily create a DB framework which makes data a joy to work with. It may even be that writing our own engine from scratch is the best way to go, all things considered.

The criteria we should keep in mind is (on top of my head):

  • Support for browser engines
  • Support remote DB providers
  • JSON intermediate format
  • Easy to implement custom data providers
  • Future proof and adaptable

Browser based engines

The first point on this list, support for browser engines, is questionable. Smart Mobile Studio ships with support for WebSQL and IndexDB, and while these are supported natively by most browsers – they serve little or no purpose beyond storing name/value pairs. And they are both crippled by a ridicules 5-10 megabyte limitation.

This really is the sad truth, that browser environments at this point in time is pretty much useless when it comes to data storage. The reality is that native engines provided by the browser, gives you absolutely no benefit over a hand-sculpted, structured, indexed array. You just dont gain anything useful by using these DB engines.

SQLite

Smart Mobile Studio ships with a third engine, one which IMHO is a much better alternative than any of the native, browser based options. We took the time to compile the now industry standard SQLite from it’s orginal C source code, into JavaScript. Yes you read right, we have a pure JavaScript version of SQlite  which does everything it’s native counterpart is capable of, including creating binary compatible DB files.

But before you say “Great! Let’s use that”, keep in mind the other points on our list, namely “future proof” and “support remote DB providers”. If we marry SQLite by tooth and claw, we may end up with serious problems in the future. The database capabilities of the browser will change, just like object pascal did years ago. Remember when Paradox was the default database? Remember how hard it was to get rid of the BDE only 10 years ago? Well, if we can avoid that kindof scenario we should. By any means.

In my view SQLite should be an engine, one you can chose to use for local data storage if you need to work with complex data structures (read: data that needs SQL to work properly). It should just be one solution in a series of options, ranging from TClientDataset to TRemoteDataset (or something in that ballpark).

So already we begin to sense that our framework must abstract away any notion of where data comes from. Also, since JavaScript is purely ASYNC (including network capabilities), the link between provider and consumer must have a signaling mechanism. Requesting data and consuming data does not occur in sequence, like we have under Delphi. So we can write that down and keep it in mind as we dig deeper.

Some benefits

The benefit of picking just one engine, like SQLite, is that you can write code that pretty much knows what is going to happen. If the only engine you will ever support is SQLite then naturally your code will reflect that and you dont have to take height for scenarios which SQLite doesnt cause. So there are benefits to sticking to a simple, ad-hoc architecture. The downside is that – once data binding get’s into the codebase, throwing it out again or altering it in the future is going to be a nitemare. It may even break the codebase completely, especially for our customers.

Complete abstraction

In a perfect world data-consumers and data-providers doesnt know “how” things are done, or even what format data comes in. They only know about the access interface between them and that’s it.

From what we have covered so far, it becomes clear that in order to deal with multiple sources of data (local database, remote access) that at least on this level, abstraction is a must.

In my mind, at least 3 source types should be supported out of the box:

  • Remote DB
  • Local DB (WebSQL, SQLite drivers, TClientDataset)
  • UDD (User defined data)

What we end up with is an architcture very much like the .net framework. A framework which involves logical, linear naming as such:

  • DataEngine (*1)
    • DataConnection (*1)
  • Dataprovider
  • Dataset
  • Datasource
  • Databinding
    • DataEmitter (*2)
    • DataConsumer (*2)

*1) A data connection object contains properties and methods for executing and maintaining a connection to a remote host. Typically this object is exposed as a property of a DataProvider, which in turn represents a particular engine.

*2) Internal mechanisms used by visual components as couplings to a data-binding. The binding itself does little but describe source and target, while emitter and consumer objects are created on each side of the binding when the binding is activated

Data provider

What is a data provider? It is an object which provides access to structured data. It can be a table, the result of a query, a fixed user defined list of values or anything that can be represented as a linear, structured set of data.

Dataset

Like the name implies this is a set of data, a collection of items or rows -where each row has a series of columns, and each column can contain data of a particular type.

Datasets are exposed by dataproviders, they can be bound or unbound. Bound datasets are live and any update/write operation is performed on the database itself (with corresponding update to any DB aware components). Unbound datasets have no such contract and must be manually updated.

Data source

This is the bridge between the non-visual and visual components. A data-source keeps track of a collection of data-bindings and replicates update signals to attached components.

Databinding

A databinding is an object which describes a connection between a visual control (textbox, boolean switch etc.) and a dataset field. Whenever the visual control alter the value, the change also occurs in the datasource. And whenever the value is altered in the datasource, the databinding makes sure the value is also updated in any bound visual controls.

When a databinding is activated (for instance when a dataset is populated by data, like a table being opened), it creates two auxillary objects: A data emitter and a data consumer object. The emitter and consumer is created as child objects for the binding itself, but the visual control is given an interface to the consumer so it can read and write data through that.

Visual programming, extending the IDE

All of the above is great and it will no doubt give us the same benefits as .net developers currently enjoy. But in order for databases to become a living part of the Smart Mobile Studio cycle, we need to extend the IDE.

The first option is to do like Delphi does it, and allow non-visual objects to be dropped onto a form. But this is truly a complete waste of CPU power and to be honest. It can be done (actually quite easy), but I have nothing positive to say about it. It opens up for a world of pain. People are lazy and will copy stuff between forms rather than isolate data access in one, central place. Trust me on this, it would be to take a step backwards.

A much better idea is to create a new entity. A visual component, but one that can only be dropped onto a special area of the form designer. This gives us plenty of benefits:

  • We avoid the overhead of TComponent
  • All objects can be created by the automatically generated code
  • Lightweight and fast, avoid code bloat
  • Easier for us to implement in the IDE
  • Easier to add to the existing package format

Here is a picture (example only, ignore colors) of what I have in mind. As you can see we have a bottom horizontal region where non-visual components can be dropped. When you select them you get to edit their properties in the property-inspector, just like under Visual Studio.

Globals and locals

In the above idea we find an interesting concept. First is the obvious, namely that the generated startup-code for a form creates the non-visual components when the form is created. But we can also introduce a global scope drag & drop placeholder (!)

dragAndDrop

A non visual drag & drop area like Visual Studio has would be nice

The global placeholder, which becomes available when you select the application unit, would be automatically created when the application starts. This is actually very powerful. It means that we can isolate our central DB components here, and they are compiled into TApplication. They would be globally available to the entire application.

The benefit is that on form level, all we have to drop is a datasource and then X number of bindings linking to visual components. See?

And it doesnt stop there, once we have stuff like image-lists implemented, they too can be make globally available by keeping them in “globals”. I know it’s not the same as a TDatamodule, but when it comes to DB programming it’s actually a restriction that helps us write cleaner code. You wont believe some of the horrible DB code i have seen over the years. People copying queries into every form, bloating the application and abusing the database, rather than isolating all data access in a single datamodule.

Your ideas matter

I would very much like to hear your ideas around this. And please, spend a few minutes reflecting on the model. And if you have a model presently in use that you feel SMS would benefit from – then dont hessitate to present it. The goal here is to create a solution which is future proof, easy to work with and adaptable. Not “who is the coolest programmer”.

I also think it’s important to have a closely knit, open and friendly community — hence i share and write as much as I do. I’ve never been a fan of closed doors or companies that keep their friends at arms-length.

Well, have a happy XMas and let’s make 2016 a kick ass Smart Mobile Studio year!

 

Advertisements
  1. December 26, 2015 at 7:37 pm

    Why wouldn’t you post this on the SMS website news page? That is where your customers are.

    • Jon Lennart Aasenden
      December 26, 2015 at 8:37 pm

      Because this is my personal blog where i write personal ideas. The blog will shift to the official website over xmas

  2. December 26, 2015 at 8:22 pm

    Perhaps you may simply use an existing Open Source approach, like the cross-platform client available in our mORMot framework. It would be IMHO easier for us to add client-side storage in our own code (which already has JSON, REST, SOA, and ORM built-in, not only for SMS, but also for FPC and Delphi).
    Focus on the IDE and the compiler. Allow easy UI development for mobile, e.g. using an open source framework like “Framework 7” or “Ionic” – only the CSS part, since SMS does wonders for the MVC/Data part. See e.g. what Warleyalex made with F7 + SMS: https://rawgit.com/smartpascal/smartms/master/smsbasic/www/index.html

    • December 27, 2015 at 8:19 am

      I absolutely agree, Open Source is the only way not let down pascal, and bring to platform some “fresh blood”, ideas, actions and so on.

    • Jon Lennart Aasenden
      December 27, 2015 at 11:12 am

      A Bouchez: I have tried to contact you on several occations, but sadly it’s quite hard to locate you on skype or google chat. If you would be so kind as to send me an email so we could get in touch? jon.lennart.aasenden AT gmail.com ?

  3. December 26, 2015 at 9:01 pm

    About the “global” repository/datamodule, it is a RAD idea…. It clearly breaks the SOLID principles to have the UI so coupled with the DB/data, using a single global content. For a pure UI client application, with a single execution context, it may be safe in most simpler cases. But for a server application (when running e.g. on node.js), it would be a real PITA. Even for a client application, such a global container is IMHO unsafe. Asynchronous (JavaScript) calls may have some time collision (especially when executing remote or local services), which may break this global repository continuity state… and induce unexpected issues, very difficult to reproduce, track, and fix.

    • Jon Lennart Aasenden
      December 27, 2015 at 12:12 am

      Well, a global scope collection of objects (e.g “datamodule”) would not be that bad. First of all it wont be truly global, but rather the objects would be created as part of TApplication. So it would make use of partial classes to achieve this. So under javascript the objects would still be created exclusively within the scope of the application. Which is embedded within $APPNAME namespace.

      The same would apply to nodeJS, but there we face another challenge when it comes to non-visual components, because NodeJS is a completely different cup of tea all together. So while some non-visual objects can be shared between a visual smart app and nodeJS, it would be very limited.
      TW3Dataset and other, pure object pascal, classes would work just fine, but anything dealing with graphics for instance would have to be re-implemented.

  4. December 28, 2015 at 3:34 am

    Hi Jon
    Thanks for the opportunity to leave comments / have a conversation about the future of SMS.

    I suppose there is a fine line between adding features to the core of SMS on the one hand side and figuring out how to leverage external facilities on the other. If you’re adding features which are readily available elsewhere you’ll end up with bloatware but some featurers like db-access are so fundamental it probably is a necessity to include them in the core.

    My personal list of ‘what do I need’ is

    – DB access.
    I’m quite happy with accessing simple MySQL databases on the back-end through ajax/httprequests and php-scripts. Works fine for simple set-ups where the number of users is limited. However, applications with larger number of users where concurrency control etc is paramount needs a more robust DB-access mechanism.

    Isn’t a DB-access mechanism in the end not always based on a 3-tier system where the database is tier 1, the client is tier 3 and in the middle (2) there is either a server or a proprietary library which performs that function ?
    Not sure if you’re incorporating in your thinking this middle layer and IMHO this is the most critical / difficult part ?

    So we need to be able to access MySQL, Oracle, DB2 etc. server-side in a multi-device / multi-user scenario with access to all features of these DB’s (transaction logging, concurrency control, sync-points, roll-backs etc)
    (By the way, I have no experience with NoSQL / Hadoop environments which are tailored for really large and distributed datasets, but lets not exclude them from the equation).

    Databases hosted on a private/company DB server is one use-case. In other circumstances it may be worthwhile involving external DB-service providers like Amazon / Google / Microsoft and it would be great if we can use SMS to hook into those services.

    – API’s
    While it is great to be able to do things natively (like image handling) it is in many cases worthwhile to see if there are other frameworks or services out there that we can use or hook in to. I posted a story on how to use Cloudinary for image management from SMS a while ago as an example.

    Other examples I have in mind will be to see if we can hook into Watson / Bluemix. How good would it be to be able to build an application where the customer care part is taken care of by Watson in natural language ? And this service is available now, just have to figure out how to use it.

    Same with some of the Google open-source frameworks like tensor flow for instance.

    Or to put the horse back in front of the carriage : I don’t see that facilities like deep learning or virtual reality processing will ever be built to any degree of completeness in the core of SMS. However we should strive to be able to hook into external libraries or services as they become available.

    – and lastly How-to’s
    The SMS ecosystem has many users quite divers in background and knowledge. While it is ok to figure out a lot of things yourself, its is time-consuming and I would find a more extensive collection of how-to’s very rewarding. Your blogs fill this void to some extent, but a recipe / article on how to wrap external js libraries, how to augment Googles page speed indicators etc would be very welcome.

    Cheers

    Nico

  5. December 28, 2015 at 6:29 am

    As SMS is mimicking Delphi and its RAD the data access part is way too complicated. Of course those who have not used any other dev tool except Delphi will not feel that but those coming from other dev tools will surely feel this overtly complicated and confusing to an extent.

    Need to simplify this to just one to two components instead of:

    DataEngine (*1)
    DataConnection (*1)
    Dataprovider
    Dataset
    Datasource
    Databinding
    DataEmitter (*2)
    DataConsumer (*2)

    • Jon Lennart Aasenden
      December 28, 2015 at 10:34 am

      SMS doesnt really mimic Delphi. It has some obvious similarities, and yes – since its written in Delphi we naturally want to port over as much as we can (within legal bounds), but we are free to avoid the pitfalls of Delphi.

      Reducing the number of components would severely restrict the use. For instance, how will you distribute change notification to say, 10 edit components, connected to various fields in a database without a Datasource?
      Also, you must remember NodeJS. Under Node we have native drivers for MySQL, SQLite, Oracle, MSSQL — and these must all be wrapped and isolated as TDataEngine objects.
      But under HTML5 only WebSQL, TW3Dataset and SQLite (compiled to JS) will be available.

      But in the future, we will no doubt wrap Amazon and Azure data services, so you can connect directly to your cloud database. How do you intend to support all these features without inheritance and isolation in various components?

  6. December 28, 2015 at 11:30 am

    I have already mention that the current approach to create non/visual visual objects to be dropped onto a form is truly a complete a waste of CPU power to be honest. It opens up for world of pain. I’ll prefer hand-written code to create the views, I consider to be more readable, more maintainable and believe to offer more customizations and abstractions.

    Anyway, a much better idea and I feel it would be perfect for global adoption, consider to use Framework7 (MIT license) – only the CSS part. Creating components from the ground up based on F7 would be awesome, this will give us plenty of benefits.

  1. No trackbacks yet.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: