Archive

Archive for the ‘Smart Mobile Studio’ Category

Amibian.js under the hood

December 5, 2018 2 comments

Amibian.js is gaining momentum as more and more developers, embedded systems architects, gamers and retro computer enthusiasts discover the project. And I have to admit I’m pretty stoked about what we are building here myself!

intro

In a life-preserver no less ūüėÄ

But, with any new technology or invention there are two common traps that people can fall into: The first trap is to gravely underestimate a technology. JavaScript certainly invites this, because only a decade ago the language was little more than a toy. Since then JavaScript have evolved to become the most widely adopted programming language in the world, and runtime engines like Google’s V8 runs JavaScript almost as fast as compiled binary code (“native” means machine code, like that produced by a C/C++ compiler, Pascal compiler or anything else that produces programs that run under Linux or Windows).

It takes some adjustments, especially for traditional programmers that havent paid attention to where browsers have gone – but long gone are the days of interpreted JavaScript. Modern JavaScript is first parsed, tokenized and compiled to bytecodes. These bytecodes are then JIT compiled (“just in time”, which means the compilation takes place inside the browser) to real machine-code using state of the art techniques (LLVM). So the JavaScript of 2018 is by no means the JavaScript of 2008.

The second trap you can fall into – is to exaggerate what a new technology can do, and attach abilities and expectations to a product that simply cannot be delivered.¬†It is very important to me that people don’t fall into either trap, and that everyone is informed about what Amibian.js actually is and can deliver – but also what it wont deliver. Rome was not built-in a day, and it’s wise to study all the factors before passing judgement.

I have been truly fortunate that people support the project financially via Patreon, and as such I feel it’s my duty to document and explain as much as possible. I am a programmer and I often forget that not everyone understands what I’m talking about. We are all human and make mistakes.

Hopefully this post will paint a clearer picture of Amibian.js and what we are building here. The project is divided into two phases: first to finish Amibian.js itself, and secondly to write a Visual Studio clone that runs purely in the browser. Since it’s easy to mix these things up, I’m underlining this easy – just in case.

What the heck is Amibian.js?

Amibian.js is a group of services and libraries that combined creates a portable operating-system that renders to HTML5. A system that was written using readily available web technology, and designed to deliver advanced desktop functionality to web applications.

The services that make up Amibian.js was designed to piggyback on a thin Linux crust, where Linux deals with the hardware, drivers and the nitty-gritty we take for granted. There is no point trying to write a better kernel in 2018, because you are never going to catch up with Linus Torvalds. It’s must more interesting to push modern web technology to the absolute limits, and build a system that is truly portable and distributed.

smart_ass

Above: Amibian.js is created in Smart Pascal and compiled to JavaScript

The service layer is written purely in node.js (JavaScript) which guarantees the same behavior regardless of host platform. One of the benefits of using off-the-shelves web technology is that you can physically copy the whole system from one machine to the other without any changes. So if you have a running Amibian.js system on your x86 PC, and copy all the files to an ARM computer – you dont even have to recompile the system. Just fire up the services and you are back in the game.

Now before you dismiss this as “yet another web mockup” please remember what I said about JavaScript: the JavaScript in 2018 is not the JavaScript of 2008. No other language on the planet has seen as much development as JavaScript, and it has evolved from a “browser toy” – into the most important programming language of our time.

So Amibian.js is not some skin-deep mockup of a desktop (lord knows there are plenty of those online). It implements advanced technologies such as remote filesystem mapping, an object-oriented message protocol (Ragnarok), RPCS (remote procedure call invocation stack), video codec capabilities and much more — all of it done with JavaScript.

In fact, one of the demos that Amibian.js ships with is Quake III recompiled to JavaScript. It delivers 120 fps flawlessly (browser is limited to 60 fps) and makes full use of standard browser technologies (WebGL).

utube

Click on picture above to watch Amibian.js in action on YouTube

So indeed, the JavaScript we are talking about here is cutting edge. Most of Amibian.js is compiled as “Asm.js” which means that the V8 runtime (the code that runs JavaScript inside the browser, or as a program under node.js) will JIT compile it to highly efficient machine-code.

Which is why Amibian.js is able to do things that people imagine impossible!

Ok, but what does Amibian.js consist of?

Amibian.js consists of many parts, but we can divide it into two categories:

  • A HTML5 desktop client
  • A system server and various child processes

These two categories have the exact same relationship as the X desktop and the Linux kernel. The client connects to the server, invokes procedures to do some work, and then visually represent the response This is identical to how the X desktop calls functions in the kernel or one of the Linux libraries. The difference between the traditional, machine code based OS and our web variation, is that our version doesn’t have to care about the hardware. We can also assign many different roles to Ambian.js (more about that later).

smartdesk

Enjoying other cloud applications is easy with Amibian.js, here is Plex, a system very much based on the same ideas as Amibian.js

And for the record: I’m¬†trying to avoid a bare-metal OS, otherwise I would have written the system using a native programming language like C or Object-Pascal. So I am not using JavaScript because I lack skill in native languages, I am using JavaScript because native code is not relevant for the tasks Amibian.js solves. If I used a native back-end I could have finished this in a couple of months, but a native server would be unable to replicate itself between cloud instances because chipset and CPU would be determining factors.

The Amibian.js server is not a single program. The back-end for Amibian.js consists of several service applications (daemons on Linux) that each deliver specific features. The combined functionality of these services make up “the amibian kernel” in our analogy with Linux. You can think of these services as the library files in a traditional system, and programs that are written for Amibian.js can call on these to a wide range of tasks. It can be as simple as reading a file, or as complex as registering a new user or requesting admin rights.

The greatest strength of Amibian.js is that it’s designed to run clustered, using as many CPU cores as possible. It’s also designed to scale, meaning that it will replicate itself and divide the work between different instances. This is where things get’s interesting, because an Amibian.js cluster doesn’t need the latest and coolest hardware to deliver good performance. You can build a cluster of old PC’s in your office, or a handful of embedded boards (ODroid XU4, Raspberry PI’s and Tinkerboard are brilliant candidates).

But why Amibian.js? Why not just stick with Linux?

That is a fair question, and this is where the roles I mentioned above comes in.

As a software developer many of my customers work with embedded devices and kiosk systems. You have companies that produce routers and set-top boxes, NAS boxes of various complexity, ticket systems for trains and busses; and all of them end up having to solve the same needs.

What each of these manufacturers have in common, is the need for a web desktop system that can be adapted for a specific program. Any idiot can write a web application, but when you need safe access to the filesystem, unified API’s that can delegate signals to Amazon, Azure or your company server, things suddenly get’s more complicated. And even when you have all of that, you still need a rock solid application model suitable for distributed computing. You might have 1 ticket booth, or 10.000 nation wide. There are no systems available that is designed to deal with web-technology on that scale. Yet ūüėČ

Let’s look at a couple of real-life scenarios that I have encountered, I’m confident you will recognize a common need. So here are some roles that Amibian.js can assume and help deliver a solution rapidly. It also gives you some ideas of the economic possibilities.

Updated: Please note that we are talking javascript here, not native code. There are a lot of native solutions out there, but the whole point here is to forget about CPU, chipset and target and have a system floating on top of whatever is beneath.

  • When you want to change some settings on your router – you login to your router. It contains a small apache server (or something similar) and you do all your maintenance via that web interface. This web interface is typically skin-deep, annoying to work with and a pain for developers to update since it’s connected to a native apache module which is 100% dependent on the firmware. Each vendor end up re-inventing the wheel over and over again.
  • When you visit a large museum notice the displays. A museum needs to display multimedia, preferably on touch capable devices, throughout the different exhibits. The cost of having a developer create native applications that displays the media, plays the movies and gives visual feedback is astronomical. Which is why most museums adopt web technology to handle media presentation and interaction. Again they re-invent the wheel with varying degree of success.
  • Hotels have more or less the exact same need but on a smaller scale, especially the larger hotels where the lobby have information booths, and each room displays a web interface via the TV.
  • Shopping malls face the same challenge, and depending on the size they can need anything from a single to a hundred nodes.
  • Schools and education spend millions on training software and programming languages every year. Amibian.js can deliver both and the schools would only pay for maintenance and adaptation – the product itself is free. Kids get the benefit of learning traditional languages and enjoying instant visual feedback! They can learn Basic, Pascal, JavaScript and C. I firmly believe that the classical languages will help make them better programmers as they evolve.

You are probably starting to see the common denominator here?

They all need a web-based desktop system, one that can run complex HTML5 based media applications and give them the same depth as a native operating-system; Which is pretty hard to achieve with JavaScript alone.

Amibian.js provides a rich foundation of more than 4000 classes that developers can use to write large, complex and media rich applications (see Smart Mobile Studio below). Just like Linux and Windows provides a wealth of libraries and features for native application development – Amibian.js aims to provide the same for cloud and embedded systems.

And as the name implies, it has roots in the past with the machine that defined multimedia, namely the Commodore Amiga. So the relation is more than just visually, Amibian.js uses the same system architecture – because we believe it’s one of the best systems ever designed.

If JavaScript is so poor, why should we trust you to deliver so much?

First of all I’m not selling anything. It’s not like this project is something that is going to make me a ton of cash. I ask for support during the development period because I want to allocate proper time for it, but when done Amibian.js will be free for everyone (LGPL). And I’m also writing it because it’s something that I need and that I havent seen anywhere else. I think you have to write software for yourself, otherwise the quality wont be there.

Secondly, writing Amibian.js in raw JavaScript with the same amount of functions and depth would take years. The reason I am able to deliver so much functionality quickly, is because I use a compiler system called Smart Mobile Studio. This saves months and years of development time, and I can use all the benefits of OOP.

Prior to starting the Amibian.js project, I spent roughly 9 years creating Smart Mobile Studio. Smart is not a solo project, many individuals have been involved – and the product provides a compiler, IDE (editor and tools), and a vast run-time library of pre-made classes (roughly 4000 ready to use classes, or building-blocks).

amibian_shell

Writing large-scale node.js services in Smart is easy, fun and powerful!

Unlike other development systems, Smart Mobile Studio compiles to JavaScript rather than machine-code. We have spent a great deal of time making sure we could use proper OOP (object-oriented programming), and we have spent more than 3 years perfecting a visual application framework with the same depth as the VCL or FMX (the core visual frameworks for C++ builder and Delphi).

The result is that I can knock out a large application that a normal JavaScript coder would spend weeks on – in a single day.

Smart Mobile Studio uses the object-pascal language, a dialect which is roughly 70% compatible with Delphi. Delphi is exceptionally well suited for writing large, data driven applications. It also thrives for embedded systems and low-level system services. In short: it’s a lot easier to maintain 50.000 lines of object pascal code, than 500.000 lines of JavaScript code.

Amibian.js, both the service layer and the visual HTML5 client application, is written completely using Smart Mobile Studio. This gives me as the core developer of both systems a huge advantage (who knows it better than the designer right?). I also get to write code that is truly OOP (classes, inheritance, interfaces, virtual and abstract methods, partial classes etc), because our compiler crafts something called a VMT (virtual method table) in JavaScript.

Traditional JavaScript doesn’t have OOP, it has something called prototypes. With Smart Pascal I get to bring in code from the object-pascal community, components and libraries written in Delphi or Freepascal – which range in the hundreds of thousands. Delphi alone has a massive library of code to pick from, it’s been a popular toolkit for ages (C is 3 years older than pascal).

But how would I use Amibian.js? Do I install it or what?

Amibian.js can be setup and used in 4 different ways:

  • As a true desktop, booting straight into Amibian.js in full-screen
  • As a cloud service, accessing it through any modern browser
  • As a NAS or Kiosk front-end
  • As a local system on your existing OS, a batch script will fire it up and you can use your browser to access it on https://127.0.0.1:8090

So the short answer is yes, you install it. But it’s the same as installing Chrome OS. It’s not like an application you just install on your Linux, Windows or OSX box. The whole point of Amibian.js is to have a platform independent, chipset agnostic system. Something that doesn’t care if you using ARM, x86, PPC or Mips as your CPU of preference. Developers will no doubt install it on their existing machines, Amibian.js is non-intrusive and does not affect or touch files outside its own eco-system.

But the average non-programmer will most likely setup a dedicated machine (or several) or just deploy it on their home NAS.

The first way of enjoying Amibian.js is to install it on a PC or ARM device. A disk image will be provided for supporters so they can get up and running ASAP. This disk image will be based on a thin Linux setup, just enough to get all the drivers going (but no X desktop!). It will start all the node.js services and finally enter a full-screen web display (based on Chromium Embedded) that renders the desktop. This is the method most users will prefer to work with Amibian.js.

The second way is to use it as a cloud service. You install Amibian.js like mentioned above, but you do so on Amazon or Azure. That way you can login to your desktop using nothing but a web browser. This is a very cost-effective way of enjoying Amibian.js since renting a virtual instance is affordable and storage is abundant.

The third option is for developers. Amibian.js is a desktop system, which means it’s designed to host more elaborate applications. Where you would normally just embed an external website into an IFrame, but Amibian.js is not that primitive. Hosting external applications requires you to write a security manifest file, but more importantly: the application must interface with the desktop through the window’s message-port. This is a special object that is sent to the application as a hand-shake, and the only way for the application to access things like the file-system and server-side functionality, is via this message-port.

Calling “kernel” level functions from a hosted application is done purely via the message-port mentioned above. The actual message data is JSON and must conform to the Ragnarok client protocol specification. This is not as difficult as it might sound, but Amibian.js takes security very seriously – so applications trying to cause damage will be promptly shut down.

You mention hosted applications, do you mean websites?

Both yes and no: Amibian.js supports 3 types of applications:

  • Ordinary HTML5/JS based applications, or “websites” as many would call them. But like I talked about above they have to establish a dialog with the desktop before they can do anything useful.
  • Hybrid applications where half is installed as a node.js service, and the other half is served as a normal HTML5 app. This is the coolest program model, and developers essentially write both a server and a client – and then deploy it as a single package.
  • LDEF compiled bytecode applications, a 68k inspired assembly language that is JIT compiled by the browser (commonly called “asm.js”) and runs extremely fast. The LDEF virtual machine is a sub-project in Amibian.js

The latter option, bytecodes, is a bit like Java. A part of the Amibian.js project is a compiler and runtime system called LDEF.

patron_asm2

Above: The Amibian.js LDEF assembler, here listing opcodes + disassembling a method

The first part of the Amibian.js project is to establish the desktop and back-end services. The second part of the project is to create the worlds first cloud based development platform. A full Visual Studio clone if you like, that allows anyone to write cloud, mobile and native applications directly via the browser (!)

Several languages are supported by LDEF, and you can write programs in Object Pascal, Basic and C. The Basic dialect is especially fun to work with, since it’s a re-implementation of BlitzBasic (with a lot of added extras). Amiga developers will no doubt remember BlitzBasic, it was used to create some great games back in the 80s and 90s. It’s well suited for games and multimedia programming and above all – very easy to learn.

More advanced developers can enjoy Object Pascal (read: Delphi) or a sub-set of C/C++.

And please note: This IDE is designed for large-scale applications, not simple snippets. The ultimate goal of Amibian.js is to move the entire development cycle to the cloud and away from the desktop. With Amibian.js you can write a cool “app” in BlitzBasic, run it right in the browser — or compile it server-side and deploy it to your Android Phone as a real, natively compiled application.

So any notion of a “mock desktop for HTML” should be firmly put to the side. I am not playing around with this product and the stakes are very real.

But why don’t you just use ChromeOS?

There are many reasons, but the most important one is chipset independence. Chrome OS is a native system, meaning that it’s core services are written in C/C++ and compiled to machine code. The fundamental principle of Amibian.js is to be 100% platform agnostic, and “no native code allowed”. This is why the entire back-end and service layer is targeting node.js. This ensures the same behavior regardless of processor or host system (Linux being the default host).

Node.js has the benefit of being 100% platform independent. You will find node.js for ARM, x86, Mips and PPC. This means you can take advantage of whatever hardware is available. You can even recycle older computers that have lost mainstream support, and use them to run Amibian.js.

A second reason is: Chrome OS might be free, but it’s only as open as Google want it to be. ChromeOS is not just something you pick up and start altering. It’s dependence on native programming languages, compiler toolchains and a huge set of libraries makes it extremely niche. It also shields you utterly from the interesting parts, namely the back-end services. It’s quite frankly boring and too boxed in for any practical use; except for Google and it’s technology partners that is.

I wanted a system that I could move around, that could run in the cloud, on cheap SBC’s. A system that could scale from handling 10 users to 1000 users – a system that supports clustering and can be installed on multiple machines in a swarm.

A system that anyone with JavaScript knowledge can use to create new and exciting systems, that can be easily expanded and serve as a foundation for rich media applications.

What is this Amiga stuff, isn’t that an ancient machine?

In computing terms yes, but so is Unix. Old doesn’t automatically mean bad, it actually means that it’s adapted and survived challenges beyond its initial design. While most of us remember the Amiga for its games, I remember it mainly for its elegant and powerful operating-system. A system so flexible that it’s still in use around the world – 33 years after the machine hit the market. That is quite an achievement.

image2

The original Amiga OS, not bad for a 33-year-old OS! It was and continues to be way ahead of everyone else. A testament to the creativity of its authors

Amibian.js as the name implies, borrows architectural elements en-mass from Amiga OS. Quite simply because the way Amiga OS is organized and the way you approach computing on the Amiga is brilliant. Amiga OS is much more intuitive and easier to understand than Linux and Windows. It’s a system that you could learn how to use fully with just a couple of days exploring; and no manuals.

But the similarities are not just visual or architectural. Remember I wrote that hosted applications can access and use the Amibian.js services? These services implement as much of the original ROM Kernel functions as possible. Naturally I can’t port all of it, because it’s not really relevant for Amibian.js. Things like device-drivers serve little purpose for Amibian.js, because Amibian.js talks to node.js, and node talks to the actual system, which in turn handles hardware devices. But the way you would create windows, visual controls, bind events and create a modern, event-driven application has been preserved to the best of my ability.

But how does this thing boot? I thought you said server?

If you have setup a dedicated machine with Amibian.js then the boot sequence is the same as Linux, except that the node.js services are executed as background processes (daemons or services as they are called), the core server is initialized, and then a full-screen HTML5 view is set up that shows the desktop.

But that is just for starting the system. Your personal boot sequence which deals with your account, your preferences and adaptations – that boots when you login to the system.

When you login to your Amibian.js account, no matter if it’s just locally on a single PC, a distributed cluster, or via the browser into your cloud account — several things happen:

  1. The client (web-page if you like) connects to the server using WebSocket
  2. Login is validated by the server
  3. The client starts loading preferences files via the mapped filesystem, and then applies these to the desktop.
  4. A startup-sequence script file is loaded from your account, and then executed. The shell-script runtime engine is built into the client, as is REXX execution.
  5. The startup-script will setup configurations, create symbolic links (assigns), mount external devices (dropbox, google drive, ftp locations and so on)
  6. When finished the programs in the ~/WbStartup folder are started. These can be both visual and non-visual.

As you can see Amibian.js is not a mockup or “fake” desktop. It implements all the advanced features you expect from a “real” desktop. The filesystem mapping is especially advanced, where file-data is loaded via special drivers; drivers that act as a bridge between a storage service (a harddisk, a network share, a FTP host, Dropbox or whatever) and the desktop. Developers can add as many of these drivers as they want. If they have their own homebrew storage system on their existing servers, they can implement a driver for it. This ensures that Amibian.js can access any storage device, as long as the driver conforms to the driver standard.

In short, you can create, delete, move and copy files between these devices just like you do on Windows, OSX or the Linux desktop. And hosted applications that run inside their own window can likewise request access to these drivers and work with the filesystem (and much more!).

Wow this is bigger than I thought, but what is this emulation I hear about? Can Amibian.js really run actual programs?

Amibian.js has a JavaScript port of UAE (Unix Amiga Emulator). This is a fork of SAE (scripted Amiga Emulator) that has been heavily optimized for web. Not only is it written in JavaScript, it performs brilliantly and thus allows us to boot into a real Amiga system. So if you have some floppy-images with a game you love, that will run just fine in the browser. I even booted a 2 gigabyte harddisk image ūüôā

But Amiga emulation is just the beginning. More and more emulators are ported to JavaScript; you have Nes, SNes, N64, PSX I & II, Sega Megadrive and even a NEO GEO port. So playing your favorite console games right in the browser is pretty straight forward!

But the really interesting part is probably QEmu. This allows you to run x86 instances directly in the browser too. You can boot up in Windows 7 or Ubuntu inside an Amibian.js window if you like. Perhaps not practical (at this point) but it shows some of the potential of the system.

I have been experimenting with a distributed emulation system, where the emulation is executed server-side, and only the graphics and sound is streamed back to the Amibian.js client in real-time. This has been possible for years via Apache Guacamole, but doing it in raw JS is more fitting with our philosophy: no native code!

I heard something about clustering, what the heck is that?

Remember I wrote about the services that Amibian.js has? Those that act almost like libraries on a physical computer? Well, these services don’t have to be on the same machine — you can place them on separate machines and thus its able to work faster.

47470965_10155861938320906_4959664457727868928_n

Above: The official Amibian.js cluster, 4 x ODroid XU4s SBC’s in a micro-rack

A cluster is typically several computers connected together, with the sole purpose of having more CPU cores to divide the work on. The cool thing about Amibian.js is that it doesn’t care about the underlying CPU. As long as node.js is available it will happily run whatever service you like – with the same behavior and result.

The official Amibian.js cluster consists of 5 ODroid XU4/S SBC (single board computers). Four of these are so-called “headless” computers, meaning that they don’t have a HDMI port – and they are designed to be logged into and software setup via SSH or similar tools. The last machine is a ODroid XU4 with a HDMI out port, which serves as “the master”.

The architecture is quite simple: We allocate one whole SBC for a single service, and allow the service to copy itself to use all the CPU cores available (each SBC has 8 CPU cores). With this architecture the machine that deals with the desktop clients don’t have to do all the grunt work. It will accept tasks from the user and hosted applications, and then delegate the tasks between the 4 other machines.

Note: The number of SBC’s is not fixed. Depending on your use you might not need more than a single SBC in your home setup, or perhaps two. I have started with 5 because I want each part of the architecture to have as much CPU power as possible. So the first “official” Amibian.js setup is a 40 core monster shipping at around $250.

But like mentioned, you don’t have to buy this to use Amibian.js. You can install it on a single spare X86 PC you have, or daisy chain a couple of older PC’s on a switch for the same result.

Why Headless? Don’t you need a GPU?

The headless SBC’s in the initial design all have GPU (graphical processing unit) as well as audio capabilities. What they lack is GPIO pins and 3 additional USB ports. So each of the nodes on our cluster can handle graphics at blistering speed — but that is ultimately not their task. They serve more as compute modules that will be given tasks to finish quickly, while the main machine deals with users, sessions, traffic and security.

The 40 core cluster I use has more computing power than northern europe had in the early 80s, that’s something to think about. And the pricetag is under $300 (!). I dont know about you but I always wanted a proper mainframe, a distributed computing platform that you can login to and that can perform large tasks while I do something else. This is as close as I can get on a limited budget, yet I find the limitations thrilling and fun!

Part of the reason I have opted for a clustered design has to do with future development. While UAE.js is brilliant to emulate an Amiga directly in the browser – a more interesting design is to decouple the emulation from the output. In other words, run the emulation at full speed server-side, and just stream the display and sounds back to the Amibian.js display. This would ensure that emulation, of any platform, runs as fast as possible, makes use of multi-processing (read: multi threading) and fully utilize the network bandwidth within the design (the cluster runs on its own switch, separate from the outside world-wide-web).

I am also very interested in distributed computing, where we split up a program and run each part on different cores. This is a topic I want to investigate further when Amibian.js is completed. It would no doubt require a re-design of the LDEF bytecode system, but this something to research later.

Will Amibian.js replace my Windows box?

That depends completely on what you use Windows for. The goal is to create a self-sustaining system. For retro computing, emulation and writing cool applications Amibian.js will be awesome. But Rome was not built-in a day, so it’s wise to be patient and approach Amibian.js like you would Chrome OS. Some tasks are better suited for native systems like Linux, but more and more tasks will run just fine on a cloud desktop like Amibian.js.

Until the IDE and compilers are in place after phase two, the system will be more like an embedded OS. But when the LDEF compiler and IDE is in place, then people will start using it en-mass and produce applications for it. It’s always a bit of work to reach that point and create critical mass.

tomes

Object Pascal is awesome, but modern, native development systems are quite demanding

My personal need has to do with development. Some of the languages I use installs gigabytes onto my PC and you need a full laptop to access them. I love Amibian.js because I will be able to work anywhere in the world, as long as a browser and normal internet line is available. In my case I can install a native compiler on one of the nodes in the cluster, and have LDEF emit compatible code; voila, you can build app-store ready applications from within a browser environment.

 

I also love that I can set-up a dedicated platform that runs legacy applications, games – and that I can write new applications and services using modern, off the shelve languages. And should a node in the cluster break down, I can just copy the whole system over to a new, affordable SBC and keep going. No super expensive hardware to order, no absurd hosting fees, and finally a system that we all can shape and use in a plethora of systems. From a fully fledged desktop to a super advanced NAS or Router that use Amibian.js to give it’s customers a fantastic experience.

And yes, I get to re-create the wonderful reality of Amiga OS without the absurd egoism that dominates the Amiga owners to this day. I don’t even know where to begin with the present license holders – and I am so sick of the drama that rolling my own seemed the only reasonable path forward.

Well — I hope this helps clear up any misconceptions about Amibian.js, and that you find this as interesting as I do. As more and more services are pushed cloud-side, the more relevant Amibian.js will become. It is perfect as a foundation for large-scale applications, embedded systems — and indeed, as a solo platform running on embedded devices!

I cant wait to finish the services and cluster this sucker on the ODroid rack!

If you find this project interesting, head over to my Patreon website and get involved! I could really use your support, even if it’s just a $5 “high five”. Visit the project at:¬†http://www.patreon.com/quartexNow

Admin woes on Delphi Developer

November 17, 2018 8 comments

For well over 10 years I have been running different interest groups on Facebook. While Delphi Developer is without a doubt the one that receives most attention from myself and my fellow moderators, I also run the Quartex Components group and lately, Amiga Disrupt. The latter dedicated to my favorite hobby, namely retro computing.

I have to say, it’s getting harder to operate these groups under the current Facebook regime. I applaud them for implementing a moral codex, that is both fair and good, but that also means that their code must be able to distinguish between random acts of hate and bullying, and moderator operations.

A couple of days ago I posted an update picture from Amibian.js. This is a picture of my vmware development platform, with pascal code, node.js and the HTML5 desktop running. You would  have be completely ignorant of technology to not recognize the picture as having to do with software development.

amibian_shell

This picture was flagged as hateful, and was enough to get an admin’s account frozen for 30 days

Sadly facebook contains all sorts of people, and for some reason even grown men will get into strange, ideological debates about what constitutes retro-computing. In this case the user was a die-hard original-amiga fan, who on seeing my post about amibian.js went on a spectacular rant. Listing in alphabetical and chronological order, the depths of depravity that people have stooped to in implementing 68k as Javascript.

Well, I get 2-3 of these comments a week and the rules for the group is crystal clear: if you post comments like that, or comments that are racist, hateful or otherwise regarded as a provocative to the general group standard — you are given a single warning and then you are out.

So I gave him a warning that such comments are not welcome; He immediately came back with a even worse response – and that was the end of that.

But before I managed to kick the user, he reported a picture of Amibian as hateful. Again, we are talking about a screen-dump from VMWare with pascal code. No hate, no poor choice of images – nothing that would violate ordinary Facebook standards.

The result? Facebook has now frozen my account for 30 days (!)

Well I’m not even going to bother being upset, because this is not the first time. When people seem to willfully seek out conflict, only to use the FB’s reporting tools as weapons of revenge — well, there is not much I can do.

Anyways, Gunnar, Glenn, Peter and Dennis got you covered – and I’ll see you in a month. I think it’s time i contact FB in Oslo and establish separate management profiles.

Leaving The Smart Company

October 30, 2018 Leave a comment

Effective immediately (30.10.2018) I am leaving The Smart Company AS and I have re-distributed my shares.

It’s almost unreal to think that it’s close to nine years since I started this project. Smart Mobile Studio continues to be a technology I am passionate about, and I must admit this is a tough call. It has taken me months to arrive at this decision, but I sadly see no other alternative given the circumstances.

In retrospect, we probably released the technology too early. I see more and more Delphi and C++ builder developers waking up to JavaScript and what web technology can do in the right hands. In other words, they are now where I was nine years ago.

When it comes to reasons there is not really much to say. There have been a few internal issues that were unfortunate, but for me this boils down to time, money and vision. Not really anything juicy to share, im simply not interested in being a partner under the terms the board currently operates with, not because I don’t believe in the product, but because I find the modus operandi counter productive.

Having said that, I am thankful for the journey and everything I have learned, and wish the team all the best for the future.

Smart Mobile Studio lives on

Even though I’m leaving the company and am re-distributed my stock, the product will continue without me. I still use and will continue to use Smart Mobile Studio in my work. But I no longer represent the company, nor will I be involved in further development. So my¬†role as head of research and development is over.

smart_ass

My new compiler core and web IDE is written in Smart Pascal

There is a time and place for all things, and while it breaks my heart to hand Smart Mobile Studio over to a future without me; my time right now is better spent at Embarcadero – working to promote and deliver the language I love above all else; namely Delphi.

Besides Embarcadero I do consulting and occasional training sessions. I have also taken on responsibilities connected with my Patreon project. So I have more than enough to keep me occupied, both at Embarcadero and personally.

But this is not a clean cut. There is no animosity involved. I will continue to use Smart Mobile Studio to build cool stuff. I will publish articles on things I make and continue to evolve the QTX Framework (which has been dormant for two years now).

Sincerely

Jon L. Aasenden

Delphi Developer Competition

September 28, 2018 Leave a comment

The Delphi Developer group on Facebook has been around for a few years, and in that time we have held two very interesting demo competitions. The last competition we held was for Smart Pascal (Smart Mobile Studio) only, but we are extending it to include the dialects supported by our group; meaning Delphi, Smart Pascal, Freepascal and Remobjects Oxygene!

Embarcadero shipped over some extra goodies for us, so the competition this year is indeed a magical one. The top 3 contestants all get the official Embarcadero T-Shirt. We also throw in 10 Sencha ball-pens for each of the top 3 contestants; this is in addition to the actual prizes listed below (!)

The #1 winner not only get the 100‚ā¨ FPGA devkit (see prizes below), he or she walks off with a high-quality, stainless steel Embarcadero branded coffee mug that holds half a litre of breakfast! (I seriously wanted to keep this for myself).

all_prices

The prizes in all their glory!

Submission rules are:

  • Source submission (GPL, LGPL) + binary
  • No dependencies on commercial libraries or components
  • Submissions must be available through GIT or BitBucket
  • Submission must include everything it needs to be compiled

Submission categories are:

  • Graphical demo (demo-scene style)
  • Games and multimedia
  • General purpose (utility programs)

Use the following Google form to register:

The purpose of the submissions is to show off both the language and your skills. Back in 2013 we got a ton of really cool demo-scene stuff, demonstrating timeless techniques; everything from bouncing meta-balls, gouraud shaded vectors, sinus scroll-texts and webgl landscape flight. We also had a fantastic fractal explorer program, bitmap rotozoom generator – and two great games! Which both made it onto AppStore and Google Play!

First prize

first_price.png

The winner walks off with some exciting stuff!

The first prize this year is something really, really special. The winner walks off with a spiffing Altera Cyclone IV FPGA starter board. This is a spectacular FPGA kit that allows you to upload a wide range of ready-to-rock FPGA core’s, as well as your own logic designs.

But to make it more accessible we added a retro daughter board, this gives you VGA, audio, keyboard, mouse, MicroSD, serial and two old school joystick ports. The daughterboard is needed if you plan on using some of the retro-cores out there. I personally love the Amiga core (shock, I know) but you can run anything from a humble Spectrum to Sega Megadrive, SNES, Atari ST/E, Neo-Geo and many others.

While the daughter-board makes this wonderful for retro-computing and gaming, fpga is first and¬†foremost a tool for engineering. It ships with a USB-Blaster which allows you to connect it directly to your PC and it will be recognized as a device. FPGA modeling applications will pick this up and you can test out designs “live”, or just place a core on the SD-card and edit the boot config.

The kit sells for roughly 100‚ā¨ with a case, but getting both the motherboard and the retro daughter-board is difficult. These things are sold separately, and the daughter board is produced in small numbers by dedicated hackers. So winning a kit that is pre-assembled, soldered and ready to go is quite a prize!

If you are even remotely interested in FPGA programming, this should give you goosebumps!

Second prize

tinker

The most powerful SBC I have ever used

The silver medal is the powerful Asus Tinkerboard, this is probably the most powerful SBC you can get below 100‚ā¨. It delivers 10 times the firepower a Raspberry PI 3b can muster – and is superbly suited for Android development, Smart Mobile Studio kiosk systems and much, much more.

Of all the board I have tested and own this is the one with enough CPU grunt (even the mighty ODroid XU4 can’t touch this) to rival a low-end x86 laptop. You have to fork out for a SnapDragon IV to beat the Tinkerboard.

I have two of these around the house myself, one as a game console running Emulation Station (emulates PSX 1, 2 and 3 games), and another under my TV with Kodi and a 2 terabyte movie collection.

Third prize

Last but not least the bronze medal is a Raspberry PI 3b. The PI should be no stranger to programmers today, it more or less defines the IOT revolution and has, by far, the biggest collection of software available of all SBC (single board computers) available today.

Raspberry_Pi_3_Large

The device that represents the IOT phenomenon

The PI is a wonderful starter board for Delphi developers who want to play with hardware under android. It’s also a fantastic board for Smart and FPC development.

I use a PI to test node.js services written in Smart Mobile Studio.

Dates

We start the clock on the 1st of october and submission must be delivered by the 31st. So you have a full month to code something cool!

Remember comments

While not always possible, try to write clean code. Part of the point here is to use these demos as an educational source.

We wont reject non-commented code, but please try to avoid 20k lines of spaghetti.

Hints and tips

Delphi has brilliant support for DirectX and OpenGL, so taking advantage of hardware acceleration should not be a problem. FMX is largely powered by the GPU and has 3d rendering and modeling as an integral feature – so Delphi developers have a slight advantage there.

16_bit_smb2_smm_wip_by_trackmasterfan341-da3nch3

Tilesets are graphics-blocks that can be used to create large game levels with a map-editor

If you want to use DIB’s under vanilla WinAPI there is always Graphics32, a wonderful and exceptionally detailed library for fast graphics.

Music: Most demo-scene code use mod music (actually today people play MP3’s as well), and there are good wrappers for player libraries like Bass. It’s always a nice touch to add a spot of music (and literally millions of free mod tracks¬†freely available). So give your demo some flair by adding a kick-ass mod track, or impress us by writing a score yourself?

In the world of demo coding anything goes! Bring out that teenage spirit and go wild, create wonderful graphical effects, vector objects, scrolling texts, games or whatever tickles your fancy. If you need inspiration, check out the demo scene videos on YouTube (if that is what you would like to submit of course). A kick-ass database application, X server renderer, paint program or a compiler — it’s all good!

Make people go WOW that is cool!

Tile graphics: which is often used in games and demos, can be found almost anywhere. If you google “tileset” or “game tiles” you should get more than you need. Brilliant for parallax scrolling. Why not give Super Mario a run for its money? Show the next generation how to code a platform game! Check out the Tiled map-editor, this has a JSON export filter for you Smart Pascal coders.

screenshot-objects

Tiled is a powerful map editor. There is also mappy, which I believe have a Delphi player

OK guys, the game is a-foot! May the best coder win!

Smart Mobile Studio presentation in Oslo

September 28, 2018 Leave a comment

Yesterday evening I traveled to Oslo and held a presentation on Smart Mobile Studio. The response was very positive and I hope that everyone who attended left with some new ideas regarding JavaScript, the direction the world of software is heading –¬†and how Smart Mobile Studio can be of service to Delphi.

Smart Pascal is especially exciting in concert with Rad-Server, where it opens the doors to Node based, platform independent services and sub clustering. With relatively little effort Rad-Server can absorb the wealth that node has to offer through Smart – but on your terms, and under Delphi’s control. The best of both worlds.

You get the stability and structure that makes Delphi so productive, and then infuse that with the flamboyance, flair and async brilliance that JavaScript represents.

More important than technology is the community! It’s been a few years since I took part in the Oslo Delphi Club’s meetups, so it was great to chat with Halvard Vassbotten,¬†Trond Gr√łntoft, Alf Christoffersen, Torgeir Amundsen and Robin Bakker face to face again. I also had the pleasure of meeting some new Delphi developers.

prespic

Presentation at¬†ABG Sundal Collier’s offices in Oslo

Thankfully the number of attendees were a moderate 14, considering this was my first presentation ever. Last time I visited was when our late PaweŇā GŇāowacki presented FMX, and the turnout was in the ballpark of a hundred. So it was an easy-going, laid-back atmosphere throughout the evening.

Conflict of interest?

Some might wonder why a person working for Embarcadero will present Smart Mobile Studio, which some still regard as competition. Smart is not in competition with Delphi and never will be. It is written by Delphi developers for Delphi developers as a means to bridge two worlds. It’s a project of loyalty and passion. We continue because we love what it enables us to do.

The talks on Smart that I am holding now, including the november talk in London, were booked before I started at Embarcadero (so it’s not a case of me promoting Smart in leu of Embarcadero). I also made it perfectly clear when I accepted the job that my work on Smart will continue in my spare time. And Embarcadero is fine with that. So I am free to spend my after-work hours and weekend time as I see fit.

smart_desktop

The Smart Desktop, codename Amibian.js, is a solid foundation for building large-scale web front-ends. Importing Sencha’s JS API’s can be done via our TypeScript wizard

So, after my presentation in London in november Smart Mobile Studio presentations (at least hosted by me) can only take place during weekends. Which is fair and the way it should be.

Recording the English version

Since the presentation last evening was in Norwegian, there was little point in recording it. Norway have a healthy share of Delphi developers, but a programming language available internationally must be presented in English.

techA couple of months back, before I started working for Embarcadero I promised to do a video presentation that would be available on Delphi Developer and YouTube. I very much like to keep that promise. So I will re-do the presentation in English as soon as possible. I would have done it today after work, but buying tech from the US have changed quite dramatically in just a couple of years.

In short: I haven’t received the remaining equipment I ordered for professional video recording and audio podcasting (which is a part of my Patreon offering as well), as such there will be no live video-feed /slash/ webinar – and questions will be limited to either the comment-section on Delphi Developer; or perhaps more appropriate, the Smart Mobile Studio Forums.

I’m hoping to get the HD camera, mic-table-arm and various bits-and-bobs i ordered from the US sometime next week. I have no idea why FedEx have become so difficult lately, but the package is apparently at LaGuardia, and I have to send receipts that document that these items are paid for before they ship them abroad (so the package manifest listing me as the customer, my address, phone number and receipt from the seller is somehow not enough). This is a first for me.

Interestingly they also stopped a package from Embarcadero with giveaways for my upcoming Delphi presentation in Sweden – at which point I had to send them a copy of my work contract to prove that I indeed work for an American company.

But a promise is a promise, so come rain or shine it will be done. Worst case scenario we can put Samsung’s claims to the test and hook up a mic + photo lens and see if their commercials have any merit.

HexLicense, Patreon and all that

September 6, 2018 Comments off

Apparently using modern service like Patreon to maintain components has become a point of annoyance and confusion. I realize that I formulated the initial HexLicense post somewhat vague and confusing, in retrospect I will admit that and also take critique for not spending a little more time on preparations.

Having said that, I also corrected the mistake quickly and clarified the situation. I feel some of the comments have been excessively critical for something that, ultimately, is a service to the community. But I’ll roll with the punches and let’s just put this issue to bed.

From the top please

fromthetopI have several products and frameworks that naturally takes time to maintain and evolve. And having to maintain websites, pay for tax and invoicing services, pay for hosting (and so on), well it consumes a lot of hours. Hours that I can no longer afford to spend (my work at Embarcadero must come first, I have a family to support). So Patreon is a great way to optimize a very busy schedule.

Today developers solve a lot of the business strain by using Patreon. They make their products open source, but give those that support and help fund the development special perks, such as early access, special builds and a more direct line of control over where the different projects and sub-projects are heading.

The public repository that everyone has access to is maintained by pushing the code on interval, meaning that the public “free stuff” (LGPL v3 license) will be some months behind the early-access that patrons enjoy. This is common and the same approach both large and small teams go about things in 2018. Quite radical compared to what we “old-timers” are used to, but that’s how things work now. I just go with flow and try to do the most amount of good on the journey.

Benefits of Patreon

The benefits are many, but first and foremost it has to do with time. Developer don’t have to maintain 3-4 websites, pay for invoicing services on said products, pay hosting fees and rent support forums — instead focus is on getting things done. So instead of an hour here and there, you can (based on the level of support) allocate X hours within a week or weekend that are continuous.

4a128ea6852444fbfc89022be4132e9b

Patreon solves two things: time and cost

Everyone wins. Those that support and help fund the projects enjoy early access and special builds. The community at large wins because the public repository is likewise maintained, albeit somewhat behind the cutting edge code patrons enjoy. And the developers wins because he or she doesn’t have to run around like a mad chicken maintaining X number of websites -wasting more time doing maintenance than building cool new features.

 

And above all, pricing goes down. By spreading the cost over a larger base of interest, people get access to code that used to cost $200 for $35. The more people that helps out, the more the cost can be reduced per tier.

To make it crystal clear what the status of my frameworks and component packages are, here is a carbon copy from HexLicense.com

For immediate release

Effective immediately HexLicense is open-source, released under the GNU Lesser General Public License v3. You can read the details of that license by clicking here.

Patreon model

Patreon_logo.svgIn order to consolidate the various projects I maintain, I have established a Patreon account. This means that people can help fund further development on HexLicense, LDEF, Amibian and various Delphi libraries as a whole. This greatly simplifies things for everyone.

I will be able to allocate time based on a broader picture, I also don’t need to pay for invoicing¬†services, web hosting and more. This allows me to continue to evolve the components and code, but without so many separate product identities to maintain.

Patreon supporters will receive updates before anyone else and have direct access to the latest code at all times. The public bitbucket repository will be updated on interval, but will by consequence be behind the Patreon updates.

Further security

One of the core goals on Patreon is the evolution of a bytecode compiler. This should be of special interest to HexLicense users. Being able to compile modules that hackers will be unable to debug gives you a huge advantage. The engine is designed so that the instruction-set can be randomized for a particular build. Making it unique for your application.

patron_asm1

The LDEF assembler prototype running under Smart Mobile Studio

Well, I want to thank everyone involved. It has been a great journey to produce so many components, libraries and solutions over the years – but now it’s time for me to cut down on the number of projects and focus on core technology.

HexLicense with the update license files will be uploaded to BitBucket shortly.

Sincerly

Jon Lennart Aasenden

 

 

Support my work on Patreon, get awesome stuff

September 2, 2018 3 comments

For well over a decade now I have tried my best to be of service to the Delphi community. I run six pascal forums on Facebook, I teach Delphi for free in my spare time and I help people solve problems, find jobs and get inspired.

“to utterly re-write the traditional development toolchain and create
a desktop environment and development studio that is unbound
by chipset, cpu and platform”

I am about to embark on the biggest journey I have ever undertaken, namely to deliver a technological platforms that combined will give both users and developers unprecedented advantages.

patreon

Support my work by becoming a patron

The challenge with new and awesome technology, is that it can be difficult to convey. The full implications of something revolutionary needs a little bit of gestation, maturity and overview before the “OMG” factor hits home. But thankfully the Delphi and Smart Pascal community is amongst the most learned, creative and innovative I have ever seen. Not to mention the Amiga retro scene that also have supported me – a group made up of hardware wizards, FPGA programmers and hackers that eat assembly code for breakfast.

I won’t dazzle you with empty promises or quick fixes. Every part of what I present here is rooted in code I have running in my lab. I hope that the doors Smart Mobile Studio have opened, the work I have done on the RTL and the products I have made – that they at least have earned me your patience; and that you will read this and see if it’s worthy of your support.

Context

When we released Smart Mobile Studio 3.0 we made a live web desktop demo to showcase some of the potential the technology has to offer. What was not mentioned was that this in fact was not a mockup or slap-dash demo intended to impress you with Quake III or the Bassoon music tracker. It has deeper roots and is a re-formation on the Quartex Desktop API that has been an essential part of Smart Mobile Studio since the beginning.

The desktop, codename Amibian.js, is actually a platform that is a part of a larger, loftier goal. One that was outlined to investors as early as 2013. Sadly I was unable to secure funds for it, despite the fact that two companies are using the prototype for kiosk and embedded systems already (city kiosk terminals in Spain running on ODroid XU4 ARM boards, and also an educational platform for schools in New Zealand).

The goal, to cut it short, is quite simply:¬†to utterly re-write the traditional development toolchain and create a desktop environment and development studio that is unbound by chipset, cpu and platform. In other words, to re-implement and build a “visual studio” environment that lives completely in the cloud, that can be accessed by any modern browser, on any operating system, anywhere in the world.

I’m not talking about Notepad or Ace here, I am talking about a complete IDE with form designer, database designer, cloud endpoints, multi language support and above all – the ability to compile and deploy both virtual and native applications through established build services. All of it JavaScript, all of it running on Node.js, Electron or HTML5.

You wont be drag & dropping components, you will be dropping entire ecosystems.

Smart Mobile Studio, new tools for a new age

When I started some eight years ago, this would have been impossible. There were no compilers that could take a complex language like object pascal or C++ and successfully express that as JavaScript. JavaScript on its own, at least compared to C++ or Delphi, is quite poor. Things we take for granted like classes, linear inheritance, virtual and abstract methods (requires a VMT), interfaces (and more) simply does not exist. There have been some advances lately of course, but JavaScript is and will always be, a prototype based runtime system.

For eight years the Smart Mobile Studio team have worked to create the ecosystem needed to make large-scale application development for JSVM (Javascript virtual machine, the browser, Phonegap, NodeJS and more) a reality. We have forged the compiler, the support code and an RTL spanning thousands of classes from scratch.

If is now possible to write JS based applications that rival native applications both in scope and complexity. This has without a doubt been one of the hardest tasks I have ever been involved in.

With Smart Mobile Studio in place and the foundation stone set – we can finally get to work on the real product. Namely a cloud forge unlike any other.

The Amibian desktop environment

The desktop platform that forms the basis of my work Рwas nicknamed Amibian due to its visual inspiration from Amiga OS 4.1, a modern but somewhat obscure operating system for PPC computers. But while there are cunning visual similarities, Amibian.js is a very different beast under the hood.

First of all Amibian.js is written from scratch to be cloud oriented. The Ragnarok message server at the heart of the system, is capable of delegating hundreds of users each dispatching high data volume simultaneously. It is a server system that is designed from scratch to be clustered, scalable and distributed.

devkit

The Ragnarok message protocol performs brilliantly, here testing IO messages live

You can run it together with the client, forming an OS much like ChromeOS, on something as small as a Tinkerboard ($70 embedded board) or scale it to a 100 node Amazon cluster. If node.js can be installed, Amibian can run. CPU or chipset is quite frankly irrelevant.

This is the foundation that the next generation IDE and compiler toolchain will be built on. A toolchain that doesn’t care if you prefer Linux, Windows, OSX or Android.

If you have a HTML5 compliant browser, you can create full-scale applications with the same level of depth as Delphi, and target 8 operating systems and more than 50 embedded devices.

What does that mean for Delphi users

Like Smart Mobile Studio, Amibian is not meant to compete with Delphi. It is designed to complement and extend Delphi – allowing Delphi developers to reach avenues where native code might be impractical or less cost-effective.

The new compiler is based around the LDEF virtual machine specification that I drafted spring 2018. It is written in Smart Pascal and runs on every system that node.js supports (which as of writing is substantial). LDEF is a bytecode specification designed to make native code generation easy. Unlike .Net or Java, LDEF is a register based virtual machine. It is a cross-section of how ARM, x86 and MC68000¬†CPU’s work in real life. It has stacks, registers, condition flags, data control, program control, absolute and relative addressing; and of course instructions that all CPUs support.

patron_asm1

The LDEF assembler is implemented completely in Smart Pascal. The picture shows the testbed with a visual coding editor. The assembler is meant to run under node.js server-side but can also be hosted on a website or post compiled into a native executable

When executing this bytecode under JavaScript, the runtime uses the subset of JavaScript called “Asm.js” out of the box. AsmJS is more mature than WebAssembly and less restrictive (modules are not sandboxed from the DOM). So to make it short: the code runs close to native courtesy of JIT optimization.

LDef is modular, meaning that parser, compiler, assembler and codegen (the part that converts bytecodes it to something else) are separate modules. Writing a WebAssembly codegen, x86 codegen or ARM codegen can be done separately without breaking the existing tooling.

patron_asm2

Having assembled the code (see picture above) the list command dumps the bytecodes to the console in readable fashion. It is then disassembled using the “dasm” command.

The LDEF prototype has been completely written in Smart Pascal, but a port is underway for Delphi and C++ builder. This gives Delphi developers the benefit of using bytecode libraries in their code. If you install Delphi server-side, you can use Amibian as a pure web front end for Delphi (!)

Create applications anywhere, on anything

Since everything is JavaScript you are no longer bound to chipset or CPU. You can set up Amibian on Amazon or Azure, an office server or an affordable, off the shelves SBC (single board computer). You can daisy chain 10 older PC’s into a cluster and get 5 more years out of the hardware; the compiler is made in JS; it doesn’t care if the real CPU is outdated. It cares about bytes and endian-ness, that’s it.

Screenshot

Early implementation of the desktop, here running native 68k (Amiga) code directly. Both x86 and PPC runtimes are now possible – the days of cloud are here

You can be on holiday in Spain armed only with an iPad and a BlueTooth keyboard, and should inspiration strike, you can login and write your application without even installing an app on your iPad. You just need a modern browser to start writing applications.

Patreon Tiers

Depending on your level of support, you get access to different parts of my work. As of writing I have 4 frameworks that is being maintained and that I want to continue to maintain for those that support me:

  • $5: High five! Support the work as a nice gesture
  • $10: Access to and support for developing my tweening library for VCL
  • $25:¬†License management for VCL and FMX, full source code access to Hexlicense and support for porting Ironwood to Delphi + a new REST based registration server
  • $35:¬†Rage libraries, get full access to the ByteRage database framework, Pixelrage graphics library and support their evolution. The timeline includes SQL and condition parsing which will not be covered by the current running tutorial. Want a clean Delphi alternative to SQLite? Well, let me make it for you.
  • $45:¬†LDEF assembler and virtual machine. Get full source code access to the Smart Pascal assembler (runs on node.js) and the Delphi port as soon as it rolls off the assembly line (pun intended). Enjoy proper documentation for instructions, bytecode format and enjoy both the native and web assembler application!¬†As a bonus, this level gives you access to video tutorials and recordings dealing with LDEF, HexLicense, Tweening and everything else.
  • $50: Amibian and Ragnarok: Amibian.js client, server and development toolchain.
    This is the motherload and you get to enjoy all of it before anyone else.

    • Full access to beta builds, updates, new features – all of it before anyone else!
    • Explore the Ragnarok client / server message API
    • Follow my video tutorials and let me help you dig into Smart Pascal and node.js
    • Ask questions and get a deeper understanding of both Smart Mobile Studio, Amibian.js and LDEF.
    • Have a front seat reserved as we unleash the power of Delphi, Smart Pascal and JavaScript on the world.
  • $100:¬†Amibian Embedded Setup:¬†For the true Amibian.js supporters! You get all the perks of previous tiers, but with an added bonus of pre-made Amibian.js disk images for the ODroid XU4 and the Asus Tinkerboard once LDEF and the IDE has been implemented.These disk images starts the Ragnarok server as a daemon (Linux Service) during the boot sequence. The system then continues booting into a full-screen webview that renders the Amibian.js desktop. There is no Linux desktop involved.
    This is by far the most cost effective setup for Kiosk and Embedded work with either a touch display or keyboard access.

    As an extra perk this version of Amibian.js contains an optimized version of uae.js (Amiga emulation) and is capable of executing ADF disks and harddisk images directly in their own window.

    With the service layer now fully developed, combined with truly platform independent compiler technology Рwe have in fact created an interesting alternative to ChromeOS. One with a minimal footprint that is cost effective and easy to expand. A system that you have full control over and can change, rebrand, modify and enjoy!

    Congratulations! You have helped bring Amibian.js and a whole new development paradigm into this world!

If this wets your appetite then head over to my Patreon site and show your support! I start shipping code to those that support me next week, so get onboard and let’s make it happen!

Final words

26229892_10155095303530906_800744220432589611_nPatreon is not the same as a kickstarter or a formal investment, I think this is important to underline. I hope however that you find my work interesting and that you would like to see this realized.

LDEF is not just a fancy bytecode runtime, it is also a framework that other developers can use to make new languages. The whole point of this is to blow the old limitations away and to really push technology to the maximum.

Being able to write system services that work the same on all operating-systems, and then deploy entire ecosystems¬†– this used to be science fiction. Now it’s not.

I want to thank those that have become patrons – it really means so much! If enough support my work I can allocate more time for implementing the tools the community needs and be of greater service to everyone.

Thank you for your time

Jon Lennart Aasenden

Graphics essentials in Smart Mobile Studio 3

August 5, 2018 Leave a comment

JavaScript and the DOM has a few quirks that can be a bit tricky for Delphi developers to instinctively understand. And while our RTL covers more or less everything, I would be an idiot if I said we havent missed a spot here and there. A codebase as large as Smart is like a living canvas; And with each revision we cover more and more of our blind-spots.

Where did TW3Image.SaveToStream vanish?

We used to have a SaveToStream method in TW3Image that took the raw DIB data (raw RGBA pixel data) and emitted that to a stream. That method was never really meant to save a picture in a compliant format, but to make it easy for game developers to cache images in a buffer and quickly draw the pixel-data to a canvas (or push it to localstorage, good if you are making a paint program). This should have been made more clear in the RTL unit, but sadly it escaped me. I apologize for that.

But in this blog-post we are going to make a proper Save() function, one that saves to a proper format like PNG or JPG. It should be an interesting read for everyone.

Resources are global in scope

Before we dig in, a few words about how the browser treats resources. This is essential because the browser is a resource oriented system. Just think about it: HTML loads everything it needs separately, things like pictures, sounds, music, css styles — all these resources are loaded as the browser finds them in the code – and each have a distinct URI (uniform resource identifier) to represent them.

So no matter where in your code you are (even a different form), if you have the URI for a resource – it can be accessed. It’s important to not mix terminology here because URI is not the same as a URL. URI is a unique identifier, an URL (uniform resource location) defines “where” the browser can find something (it can also contain the actual data).

If you look at the C/C++ specs, the URL class inherits from URI. Which makes sense.

Once a resource is loaded and is assigned an URI, it can be accessed from anywhere in your code. It is global in scope and things like forms or parent controls in the RTL means nothing to the underlying DOM.

Making new resources

When you are creating new resources, like generating a picture via the canvas, that resource doesn’t have an URI. Thankfully, generating and assigning an URI so it can be accessed is very simple — and once we have that URI the user can download it via normal mechanisms.

But the really cool part is that this system isn’t just for images. It’s also for raw data! You can actually assign a URI to a buffer and make that available for download. The browsers wont care about the content.

If you open the RTL unit SmartCL.System.pas and scroll down to line 107 (or there about), you will find the following classes defined:


  (* Helper class for streams, adds data encapsulation *)
  TAllocationHelper = class helper for TAllocation
    function  GetObjectURL: string;
    procedure RevokeObjectURL(const ObjectUrl: string);
  end;

  TW3URLObject = static class
  public
    class function  GetObjectURL(const Text, Encoding, ContentType, Charset: string): string; overload;
    class function  GetObjectURL(const Text: string): string; overload;
    class function  GetObjectURL(const Stream: TStream): string; overload;
    class function  GetObjectURL(const Data: TAllocation): string; overload;
    class procedure RevokeObjectURL(const ObjectUrl: string);

    // This cause a download in the browser of an object-url
    class procedure Download(const ObjectURL: string; Filename: string); overload;
    class procedure Download(const ObjectURL: string; Filename: string;
          const OnStarted: TProcedureRefS); overload;
  end;

The first class, TAllocationHelper, is just a helper for a class called TAllocation. TAllocation is the base-class for objects that allocate raw memory, and can be found in the unit System.Memory.Allocation.pas.
TAllocation is really central and more familiar classes like TMemoryStream expose this as a property. The idea here being that if you have a memory stream with something, making the data downloadable is a snap.

Hopefully you have gotten to know the central buffer class, TBinaryData, which is defined in System.Memory.Buffer. This is just as important as TMemoryStream and will make your life a lot easier when talking to JS libraries that expects an untyped buffer handle (for example) or a blob (more on that later).

The next class,¬†TW3URLObject, is the one that is of most interest here. You have probably guessed that TAllocationHelper makes it a snap to generate URI’s for any class that inherits from or expose a TAllocation instance (read: really handy for TMemoryStream). But TW3URLObject is the class you want.

The class contains 3 methods with various overloading:

  • GetObjectURL
  • RevokeObjectURL
  • Download

I think these are self explanatory, but in short they deliver the following:

  • GetObjectURL creates an URI for a resource
  • RevokeObjectURL removes a previously made URI from a resource
  • Download triggers the “SaveAs” dialog so users can, well, save the data to their local disk

The good news for graphics is that the canvas object contains a neat method that does this automatically, namely the ToDataUrl() function, which is a wrapper for the raw JS canvas method with the same name. Not only will it encode your picture in a normal picture format (defaults to png but supports all known web formats), it will also return the entire image as a URI encoded string.

This saves us the work of having to manually call GetObjectURL() and then invoke the save dialog.

Making some offscreen graphics

TW3Image is not meant for drawing, it’s like Delphi’s TImage and is a graphics container. So before we put a TW3Image on our form we are going to create the actual graphics to display. And we do this by creating an off-screen graphics context, assign a canvas to it, draw the graphics, and then encode the data via ToDataUrl().

To make things easier, lets use the Delphi compatible TBitmap and TCanvas classes. These can be found in SmartCL.Legacy. They are as compatible as I could make them.

  • Browsers only support 32 bit graphics, so only pf32bit is allowed
  • I havent implemented checkered, diagonal or other patterns – so bsSolid and bsClear are the only brush modes for canvas (and pen style as well).
  • Brush doesn’t have a picture property (yet), but this will be added later at some point. I have to replace the built-in linedraw() method with the Bresham algorithm for that to happen (and all the other primitives).
  • When drawing lines you have to call Stroke() to render. The canvas buffers up all the drawing operations and removes overlapping pixels to speed up the final drawing process — this is demanded by the browser sadly.

Right, with that behind us, lets create an off-screen bitmap, fill the background red and assign it to a TW3Image control.

To replicate this example please use the following recipy:

  1. Start a new “visual components project”
  2. Add the following units to the uses clause:
    1. System.Colors
    2. System.Types.Graphics
    3. SmartCL.Legacy
  3. Add a TW3Button to the form
  4. add a TW3Image to the form
  5. Save your project
  6. Double-Click on the button. This creates a code entry point for the default event, which for a button is OnClick.

Let’s populate the entry point with the following:

procedure TForm1.W3Button1Click(Sender: TObject);
var
  LBitmap:  TBitmap;
  LRect:    TRect;
begin
  LBitmap := TBitmap.Create;
  try
    LBitmap.Allocate(640, 480);
    LRect := TRect.Create(0, 0, LBitmap.width-1, LBitmap.Height-1);
    LBitmap.Canvas.Brush.Color := clRed;
    LBitmap.Canvas.FillRect(LRect);

    w3image1.LoadFromUrl( LBitmap.Canvas.ToDataURL('image/png') );

  finally
    LBitmap.free;
  end;
end;

The code above creates a bitmap, which is an off-screen (not visible) graphics context. We then set a background color to use (red) and fill the bitmap with that color. When this is done we load the picture-data directly into our TW3Image control so we can see it.

Triggering a download

With the code for creating graphics done, we now move on to the save mechanism. We want to download the picture when the user clicks the button.

offscreen

Offscreen graphics is quite fun once you know how it works

Since the image already have an URI, which it get’s when you call the ToDataURL() method, we don’t need to mess around with blob buffers and generating the URI manually. So forcing a download could not be simpler:

procedure TForm1.W3Button1Click(Sender: TObject);
var
  LBitmap:  TBitmap;
  LRect:    TRect;
begin
  LBitmap := TBitmap.Create;
  try
    LBitmap.Allocate(640, 480);
    LRect := TRect.Create(0, 0, LBitmap.width-1, LBitmap.Height-1);
    LBitmap.Canvas.Brush.Color := clRed;
    LBitmap.Canvas.FillRect(LRect);

    var LEncodedData:= LBitmap.Canvas.ToDataURL('image/png');
    w3image1.LoadFromUrl(LEncodedData);

    TW3URLObject.Download( LEncodedData, 'picture.png');

  finally
    LBitmap.free;
  end;
end;

Note: The built-in browser in Smart doesn’t allow save dialogs, so when you run this example remember to click the “open in browser” button on the execute window. Then click the button and voila — the image is downloaded directly.

Well, I hope this has helped! I will do a couple of more posts on graphics shortly because there really is a ton of cool features here. We picked heavily from various libraries when we implemented TW3Canvas and TCanvas, so if you like making games or display data – then you are in for a treat!

Nano PI Fire 3, part two

July 18, 2018 Leave a comment

If you missed the first installment of this test, please click here to catch up. In this installment we are just going to dive straight into general use and get a feel for what can and cannot be done.

Solving the power problem

pi-powerLike mentioned in the previous article, a normal mobile charger (5 volt, 2 amps) is not enough to support the nano-pi. Since I have misplaced my original PI power-supply with 5 volt / 3 amps I decided to cheat. So I plugged the power USB into my PC which will deliver as much juice as the device needs. I don’t have time to wait for a new PSU to arrive so this will have to do.

But for the record (and underlined) a proper PSU with at least 2.5 amps is essential to using this board. I suggest you order the official Raspberry PI 3b power-supply. But if you should find one with 3 amps that would be even better.

Web performance

The question on everyone’s mind (or at least mine) is: how does the Nano-PI fire 3 perform when rendering cutting edge, hardcore HTML5? Is this little device a potential candidate for running “The Smart Desktop” (a.k.a Amibian.js for those of you coming from the retro-computing scene)?

Like I suspected earlier, X (the Linux windowing framework) doesn’t have drivers that deliver hardware acceleration at all.

shot_desktop-1024x819-2-1024x819

Lubuntu is a sexy desktop no doubt there, but it’s overkill for this device

This is quite easy to test: when selecting a rectangle on the Lubuntu desktop and moving the mouse-cursor around (holding down the left mouse button at the same time) if it lags terribly, that is a clear indicator that no acceleration exists.

And I was right on the money because there is no acceleration what so ever for the Linux distribution. It struggles hopelessly to keep up with the mouse-pointer as you move it around with an active selection; something that would be silky smooth had the GPU been tasked with the job.

But, hardware acceleration is not just about the desktop. It’s not some flag you enable and it magically effect everything, but rather several API’s at either the kernel-level or immediate driver level (modules the kernel loads), each affecting different aspects of a system.

So while the desktop “2d blitting” is clearly cpu driven, other aspects of the system can still be accelerated (although that would be weird and rare. But considering how Asus messed up the Tinkerboard I guess anything goes these days).

Asking Chrome for the hard facts

I fired up Google Chrome (which is the default browser thank god) and entered the magic url:

chrome://gpu

This is a built-in page that avails a detailed report of what Chrome learns about the current system, right down to specific GPU features used by OpenGL.

As expected, there was NO acceleration what so ever. So I was quite surprised that it managed to run Amibian.js at all. Even without hardware acceleration it outperformed the Raspberry PI 3b+ by a factor of 4 (at the very least) and my particle demo ran at a whopping 8 fps (frames per second). The original Rasperry PI could barely manage 2 fps. So the Nano-PI Fire is leagues ahead of the PI in terms of raw cpu power, which is brilliant for headless servers or computational tasks.

FriendlyCore vs Lubuntu? QT for the win

Now here is a funny thing. So far I have used the Lubuntu standard Linux image, and performance has been interesting to say the least. No hardware acceleration, impressive cpu results but still – what good is a SBC Linux distro without fast graphics? Sure, if you just want a head-less file server or host services then you don’t need a beefy GPU. But here is the twist:

Turns out the makers of the board has a second, QT oriented distro called Friendly-core. And this image has OpenGL-ES support and all the missing acceleration lacking from Lubuntu.

I was pretty annoyed with how Asus gave users the run-around with Tinkerboard downloads, but they have thankfully cleaned up their act and listened to their customers. Friendly-elec might want to learn from Asus mistakes in this area.

Qtwebenginebrowser

QT has a rich history, but it’s being marginalized by node.js and Delphi these days

Alas, Friendly-core xenial 4.4 Arm64 image turned out to be a pure embedded development image. This is why the board has a debug port (which is probably awesome if you are into QT development). So this is for QT developers that want to use the board as a single-application system where they write the code on Windows or Linux, compile and it’s all transported to the board with live debugging back to the devtools they use. In other words: not very useful for non C/C++ QT developers.

Android Lolipop

2000px-Android_robot.svgI have only used Android on a pad and the odd Samsung Galaxy phone, so this should be interesting. I Downloaded the Lolipop disk image, burned it to the sd-card and booted up.

After 20 minutes with a blank screen i gave up.

I realize that some Android distros download packages ad-hoc and install directly from a repository, so it can take some time to get started; but 15-20 minutes with a black screen? The Android logo didn’t even show up — and that should be visible almost immediately regardless of network install or not.

This is really a great shame because I wanted to test some Delphi Firemonkey applications on it, to see how well it scales the more demanding GPU tasks. And yes i did try a different SD-Card to be sure it wasnt a disk error. Same result.

Back to Lubuntu

Having spent a considerable time trying to find a “wow” factor for this board, I have to just surrender to the fact that it’s just not there. . This is not a “PI” any more than the Tinkerboard is a PI. And appending “pi” to a product name will never change that.

I can imagine the Nano-PI Fire 3 being an awesome single-application board for QT C/C++ developers though. With a dedicated debug port making it a snap to transport, execute and do live debugging directly on the hardware — but for general DIY hacking, using it for native Android development with Delphi, or node.js development with Smart Mobile Studio – or just kicking back with emulators like Mame, UAE or whatever tickles your fancy — its just too rough around the edges. Which is really a shame!

So at the end of the day I re-installed Lubuntu and figure I just have to wait until Friendly-elec get their act together and issue proper drivers for the Mali GPU. So it’s $35 straight out the window — but I can live with that. It was a risk but at that price it’s not going to break the bank.

The positive thing

The Nano-PI Fire 3 is yet another SBC in a long list that fall short of its potential. Like many others they try to use the word “PI” to channel some of the Raspberry PI enthusiasm their way – but the quality of the actual system is not even close.

In fact, using PI in their product name is setting themselves up for a fall – because customers will quickly discover that this product is not a PI, which can cause some subconscious aversion and resentment.

37013365_10155541149775906_3122577366065348608_o

The Nano rendered Amibian.js running some very demanding demos 4 times as fast as the PI 3b, one can only speculate what the board could do with proper drivers for the GPU.

The only positive feature the Fire-3 clearly has to offer, is abundantly more cpu power. It is without a doubt twice as fast (if not 3 times as fast) as the Raspberry PI 3b. The fact that it can render highly demanding and complex HTML5 demos 4 times faster than the Raspberry PI 3b without hardware acceleration is impressive. This is a $35 board after all, which is the same price.

But without proper drivers for the mali, it’s a useless toy. Powerful and with great potential, but utterly useless for multimedia and everything that relies on fast 2D and 3D graphics. For UAE (Amiga emulation) you can pretty much forget it. Even if you can compile the latest UAE4Arm with SDL as its primary display framework – it wouldn’t work because SDL depends on the graphics drivers. So it’s back to square one.

But the CPU packs a punch that is without question.

Final verdict

Top the x86 UP board, left bottom a Raspberry PI 3, bottom right the ODroid XU4

There are a lot of stable and excellent options out there, take your time

I was planning to test UAE next but as I have outlined above: without drivers that properly expose and delegate the power of the mali, it would be a complete disaster. I’m not even sure it would build.

As such I will just leave this board as is. If it matures at some point that would be great, but my advice to people looking for a great SBC experience — get the new Raspberry PI 3b+ and enjoy learning and exploring there.

And if you are into Amibian.js or making high quality HTML5 kiosk / node.js based systems, then fork out the extra $10 and buy an ODroid XU4. If you pay $55 you can pick up the Asus Tinkerboard which is blistering fast and great value for money, despite its turbulent introduction.

Note: You cannot go wrong with the ODroid XU4. Its affordable, stable and fast. So for beginners it’s either the Raspberry PI 3b+ or the ODroid. These are the most mature in terms of software, drivers and stability.

Power for pennies, getting a server rack and preparing my ultimate coding environment

July 18, 2018 Leave a comment

One of the benefits of doing repairs on your house, is that during the cleanup process you come over stuff you had completely forgot about. Like two very powerful Apple blade servers (x86) I received as a present three years ago. I never got around to using them because I there was literally no room in my house for a rack cabinet.

Sure, a medium model rack cabinet isn’t that big (the size of a cabin refrigerator), but you also have to factor in that servers are a lot more noisy than desktop PCs; the older they are the more noise they make. So unless you have a good spot to place the cabinet, where the noise wont make it unbearable to be around,¬† I suggest you just rent a virtual instance at Amazon or something. It really depends on how much service coding you do, if you need to do dedicated server and protocol stress testing (the list goes on).

Power for pennies

serverrack

Sellers photo. It needs a good clean, but this kit would have set you back $5000 a decade ago; so picking this up for $400 is almost ridicules.

The price of such cabinets (when buying new ones) can be anything from $800 to $5000 depending on the capacity, features and materials. My needs for a personal server farm are more than covered by a medium cabinet. If it wasnt for my VMWare needs I would say it was overkill. But some of my work, especially with node.js and Delphi system services that should handle terabytes of raw data reliably 24/7, that demands a hard-core testing environment.

Having stumbled upon my blade servers I decided to check the local second-hand online forum; and I was lucky enough to find (drumroll) a second-hand cabinet holding a total of 10 blades for $400. So I’ll be picking up this beauty next weekend. It will be so good to finally get my blades organized. Not to mention all my SBC / Node.js cluster experiments centralized in one physical location. Far¬†away from my home office space (!)

Interestingly, it comes fitted with 3 older servers. There are two Dell web and file servers, and then a third, unmarked mystery box (i3 cpu + sata caddies so that sounds good).

It really is amazing how much cpu fire-power you can pick up for practically nothing these days. $50 buys you a SBC (single board computer) that will rival a Pentium. $400 buys you a 10 blade cabinet and 3 servers that once powered a national newspaper (!).

VMWare delights

All the blades I have mentioned so far are older models. They are still powerful machines, way more than $400 livingroom NAS would get you. So my node.js clustering will run like a dream and I will be able to host all my Delphi development environments via VMware. Which brings us neatly to the blade I am really looking forward to get into the rack.

I bought an empty server blade case back in 2015. It takes a PSU, motherboard, fans and everything else is there (even the six caddies for disks). Into this seemingly worthless metal box I put a second generation Intel i7 monster (Asus motherboard), with 32 gigabyte ram Рand fitted it with a sexy NVidia GEFORCE GTX 1080 TI.

37013365_10155541149775906_3122577366065348608_o

All my Delphi work, Smart work and various legacy projects I maintain, all in one neat rack

This little monster (actually it takes up 2 blade-spots) allows me to run VMWare server, which gives me at least 10 instances of Windows (or Linux, or OSX) at the same time. It will also be able to host and manage roughly 1000 active Smart Desktop users (the bottleneck will be the disk and network more than actual computation).

Being a coder in 2018 is just fantastic!

Things we could only dream about a decade ago can now be picked up for close to nothing (compared to the original cost). Just awesome!

 

What is new in Smart Mobile Studio 3.0

July 16, 2018 1 comment

Trying to sum up the literally thousands of changes we have done in Smart Mobile Studio the past 12 months is quite a challenge. Instead of just blindly rambling on about every little detail – I’ll try to focus on the most valuable changes; changes that you can immediately pick up and experience for yourself.

Scriptable css themes

theme_structure

A visual control now has its border and background styled from our pre-defined styles. The styles serve the same function in all themes even though they look different.

This might not feel like news since we introduced this around xmas, but like all features it has matured through the beta phases. The benefits of the new system might not be immediately obvious.

So what is so fantastic about the new theme files compared to the old css styling?

We have naturally gone over every visual control to make them look better, but more importantly – we have defined a standard for how visual controls are styled. This is important because without a theme system in place, making application “theme aware” would be impossible.

  • Each theme file is constructed according to a standard
  • A visual control is no longer styled using a single css-rule (like we did before), but rather a combination of several styles:
    • There are 15 background styles, each with a designated role
    • There are 14 borders, each designed to work with specific backgrounds
    • We have 4 font sizes to simplify what small, normal, medium and large means for a particular theme.
  • A theme file contains both CSS and Smart pascal code
  • The code is sandboxed and has no access to the filesystem or RTL
  • The code is executed at compile time, not runtime (!). So the code is only used to generate things like gradients based on constants; “scaffolding” code if you will that makes it easier to maintain and create new themes.

Optimized and re-written visual controls

Almost all our visual controls have been re-written or heavily adjusted to meet the demands of our users. The initial visual controls were originally designed as examples, following in the footsteps of mono where users are expected to work more closely with the code.

To remedy this we have gone through each control and added features you would expect to be present. In most cases the controls are clean re-writes, taking better advantage of HTML5 features such as flex-boxing and relative positions (you can now change layout mode via the PositionMode property. Displaymode is likewise a read-write property).

flexing

Flex boxing relieves controls of otherwise expensive layout chores and evenly distributes elements

Flex-boxing is a layout technique where the browser will automatically stretch or equally distribute screen real estate for child elements. Visual controls like TW3Toolbar and TW3ListMenu makes full use of this – and as a result they are more lightweight, requires no resize code and behave like native controls.

Momentum scrolling as standard

Apple have changed the rules for scrolling 3 times in the past eight years, and it’s driving HTML/JS developers nuts every time. We decided years ago that we had enough and implemented momentum scrolling ourselves written in Smart Pascal. So no matter if Apple or anyone else decides to make life difficult for developers – it wont bother us.

momentum

Momentum scrolling with indicator (or scrollbars) are now standard for all container controls and lists.

Our new TW3Scrollbox and (non visual) TW3ScrollController means that all our container and list controls supports GPU powered momentum scrolling by default. You can also disable this and use whatever default method the underlying web-view or browser has to offer.

Bi-directional Tab control

A good tab control is essential when making mobile and web applications, but making one that behaves like native controls do is quite a challenge. We see a lot of frameworks that have problems doing the bi-directional scrolling that mobile tabs do, where the headers scroll in-place as you click or touch them – and the content of the tab scroll in from either side (at the same time).

tabcontrol

Thankfully this was not that hard to implement for us, since we have proper inheritance to fall back on. JS developers tend to be limited to prototype cloning, which makes it difficult to build up more and more complex behavior. Smart enjoys the same inheritance system that Delphi and C++ uses, and this makes life a lot easier.

Google Maps control

Not exactly hard to make but a fun addition to our RTL. Very useful in applications where you want to pinpoint office locations.

google-maps-android-100664872-orig

Updated ACE coding editor

ACE is by many regarded as the de-facto standard text and code editor for JavaScript. It is a highly capable editor en-par with SynEdit in the Delphi and C++ world. This is actually the only visual control that we did not implement ourselves, although our wrapper code is substantial.

ace

Ace comes with a wealth of styles (color themes) and support for different programming languages. It can also take on the behavior of other editors like emacs (an editor as old as Unix).

We have updated Ace to the latest revision and tuned the wrapper code for speed. There was a small problem with padding that caused Ace to misbehave earlier, this has now been fixed.

The Smart Desktop, windowing framework

People have asked us for more substantial demos of what Smart Mobile Studio can do. Well this certainly qualifies. It is probably the biggest product demo ever made and represents a complete visual web desktop with an accompanying server (the Ragnarok Websocket protocol).

37013365_10155541149775906_3122577366065348608_o

The Smart Desktop showcases some of the power Smart Mobile Studio can muster

It involves quite a bit of technology, including a filesystem that uses the underlying protocol to browse and access files on the server as if they were local. It can also execute shell applications remotely and pipe the results back.

A shell window and command-line system is also included, where commands like “dir” yields the actual directory of whatever path you explore on the server.

Since the browser has no concept of “window” (except a browser window) this is fully implemented as Smart classes. Moving windows, maximizing them (and other common operations) are all included.

The Smart desktop is a good foundation for making large-scale, enterprise level web applications. Applications the size of Photoshop could be made with our desktop framework, and it makes an excellent starting-point for developers involved in router, set-top-boxes and kiosk systems.

Node.JS and server-side technology

While we have only begun to expand our node.js namespace, it is by far one of the most interesting aspects of Smart Mobile Studio 3.0. Where we only used to have rudimentary support (or very low-level) for things like http – the SmartNJ namespace represents high-level classes that can be compared to Indy under Delphi.

As of writing the following servers can be created:

  • HTTP and HTTPS
  • WebSocket and WebSocket-Secure
  • UDP Server
  • Raw TCP server

The cool thing is that the entire system namespace with all our foundation code, is fully compatible and can be used under node. This means streams, buffers, JSON, our codec classes and much, much more.

I will cover the node.js namespace in more detail soon enough.

Unified filesystem

The browser allows some access to files, within a sandboxed and safe environment. The problem is that this system is completely different from what you find under phonegap, which in turn is wildly different from what node.js operates with.

In order for us to make it easy to store information in a unified way, which also includes online services such as Azure, Amazon and Dropbox — we decided to make a standard.

filesys

The Smart Desktop shows the filesystem and device classes in action. Here accessing the user-account files on the server both visually and through our command-line (shell) application.

So in Smart Mobile Studio we introduce two new concepts:

  • Storage device classes (or “drivers”)
  • Path parsers

The idea is that if you want to save a stream to a file, there should be a standard mechanism for doing so. A mechanism that also works under node, phonegap and whatever else is out there.

For the browser we went as far as implementing our own filesystem, based on a fast B-Tree class that can be serialized to both binary and JSON. For Node.js we map to the existing filesystem methods Рand we will continue to expand the RTL with new and exciting storage devices as we move along.

Path parsers deals with how operative-systems name and deal with folders and files. Microsoft Windows has a very different system from Unix, which again can have one or two subtle differences from Linux. When a Smart application boots it will investigate what platform it’s running on, and create + install an appropriate path parser.

You will also be happy to learn that the unit System.IOUtils, which is a standard object pascal unit, is now a part of our RTL. It contains the class TPath which gives you standard methods for working with paths and filenames.

New text parser

Being able to parse text is important. We ported our TextCraft parser (initially written for Delphi) to Smart, which is a good framework for making both small and complex parsers. And we also threw in a bytecode assembler and virtual-cpu demo just for fun.

Note: The assembler and virtual cpu is meant purely as a demonstration of the low-level routines our RTL has to offer. Most JS based systems run away from raw data manipulation, well that is not the case here.

asmparse

Time to get excited!

I hope you have enjoyed this little walk-through. There are hundreds of other items we have added, fixed and expanded (we have also given the form-designer and property inspector some much needed love) – but some of the biggest changes are shown here.

For more information stay tuned and visit www.smartmobilestudio.com

Starting at Embarcadero

July 16, 2018 3 comments

My Facebook messenger App has been bombarded with questions since it became known that I now work for Embarcadero. Most of the messages are in the form of questions; what is the future of Smart Mobile Studio, will I be involved in this or that and so on.

Well those that have followed my blog over the years, or stay in touch with me via Delphi Developer on Facebook, should know by now that I don’t tip-toe around subjects; I tend to be quite direct, and even though it’s absurdly premature –let’s just grab the hot potato and get it over with.

Future of Smart Mobile Studio

Me working for Embarcadero will not change the future of Smart Mobile Studio. Smart is our baby and the whole team at The Smart Company AS will continue, like we have for the past eight years, to evolve, improve and foster Smart Pascal. So let me be absolutely clear on this: my work on Smart Mobile Studio will continue uninterrupted in my free time. Smart Pascal is a labour of love, passion and creativity.

So there is a crystal clear line between my personal and professional time.

Nor is there any potential conflict in this material, as some have openly speculated; Delphi and Smart Mobile Studio targets two fundamental different market segments; Delphi is the best native development suite money can buy, while Smart is the best development system for building mobile, cloud and large-scale JSVM (JavaScript virtual machine) infrastructures.

What people often forget, even though I have underlined it 10.000 times, is that Smart Mobile Studio is written in Delphi (!) It was created to compliment Delphi. To enrich and allow Delphi developer’s to re-apply existing skills in a new paradigm.

In many ways Smart Mobile Studio is a testament to what Delphi is capable of in the right hands.

My role at Embarcadero

It was actually Facebook that “outed” me when I changed my employment status. Instead of a silent alteration on my profile, it plastered the change in bold, underline and italic.

But writing anything about this would be premature, nor do I feel it’s my place to do so.

All I can say is that I am very excited to work for the company that makes Delphi. A product that I love and have used on a daily basis since it was first launched.

Smart Mobile Studio 3

To make those of you who have been worried about what might happen to Smart Mobile Studio as a consequence of my new path, I hope I have made you feel optimistic about the future so far. Because I am super optimistic! Seriously, this is awesome!

As I type this Smart Mobile Studio 3.0¬†beta 3 should be available via the automatic-update tool for our customers. I can’t remember a year where we have worked so hard; and we have achieved above and beyond the schedule we set back in 2017.

37013365_10155541149775906_3122577366065348608_o

Smart Mobile Studio 3.0 ~ Node.js server and desktop framework demo

I don’t know how many nights my girlfriend has found me scribbling data-paths and formulas on my home office white-board, or getting up at 03:30 at night to test some idea – but when you compare our new RTL with the previous, especially our focus on node.js, you will witness a quantum leap on quality, features and technical wealth.

Separating apples from pears

Speaking of the future; my blogging style wont change, but I will avoid mixing apples and pears. Delphi posts should be about Delphi, and Smart posts should be about Smart. I don’t think I need to explain why that is a good idea. It’s important to maintain that line between work and personal projects.

The only reason I mention both here now, is to put things to rest and make it clear for everyone that it’s all good. And it’s going to get even better.

Smart Mobile Studio 3.0 is an epic release! And Delphi is going from strength to strength. So there is a lot to be happy about! I cant even remember when object pascal developers had so many options at their disposal.

Cheers

Jon L. Aaasenden

 

Smart Mobile Studio: Q&A about v3.0 and beyond

July 1, 2018 4 comments

A couple of days back I posted a sneak-peek of our upcoming Smart Mobile Studio 3.0 web desktop framework; as a consequence my Facebook messenger app has practically exploded with questions.

smart_desktop

The desktop client / server framework is an example of what you can do in Smart

As you can imagine, the questions people ask are often very similar; so similar in fact that I will answer the hottest topics here. Hopefully that will make it easier for everyone.

If you have further questions then either ask them on our product forums or the Delphi Developer group on Facebook.

 

Generics

Yes indeed we have generics running in the labs. We havent set a date on when we will merge the new compiler-core, but it’s not going to happen until (at the earliest) v3.4. So it’s very much a part of Smart’s future but we have a couple of steps left on our time-line for v3.0 through v3.4.

RTTI access

RTTI is actually in the current version, but sadly there is a bug there that causes the code generator to throw a fit. The fix for this depends on a lot of the sub-strata in the new compiler-core, so it will be available when generics is available.

Associative arrays

This is ready and waiting in the new core, so it will appear together with generics and RTTI.

Databases

We have supported databases since day 1, but the challenge with JavaScript is that there are no “standards” like we are used to from established systems like Delphi or Lazarus.

Under the browser we support WebSQL and our own TW3Dataset. We also compiled SQLite from native C to JavaScript so we can provide a fast, lightweight SQL engine for the browser regardless of what the W3C might do (WebSQL has been deprecated but will be around for many years still).

Server side it’s a whole different ballgame. There you have drivers (or modules) for every possible database you can think of, even Firebird. But each module is implemented as the authors see fit. This is where our Database framework comes in, sets a standard, and we then inherit out classes and implement the engines we want.

This framework and standard is being written now, but it wont be introduced until v3.1 and v3.2. In the meantime you have sqlite both server-side and client-side, WebSQL and TW3Dataset.

Attributes

This question is often asked separately from RTTI, but it’s ultimately an essential part of what RTTI delivers.

So same answer: it will arrive with the new compiler-core / infrastructure.

Server-side scripting

22555292_1630289757034374_6911478701417326545_n

The new theme system in action

While we do see how this could be useful, it requires a substantial body of work to make a reality. Not only would we have to implement the whole “system” namespace from scratch since JavaScript would not be present, but we would also have to introduce a a secondary namespace; one that would be incompatible with the whole RTL at a fundamental level. Instead of going down this route we opted for Node.js where creating the server itself is the norm.

 

If we ever add server-side scripting it would be JavaScript support under node.js by compiling the V8 engine from C to asm.js. But right now our focus is not on server-side-scripting, but on cloud building-blocks.

Bytecode compilation

I implemented the assembler and runtime for our bytecode system (LDef) this winter / early spring; So we actually have the means to create a pure bytecode compiler and runtime.

But this is not a priority for us at this time. Smart Mobile Studio was made for JavaScript and while it would be cool to compile Delphi sourcecode to portable bytecodes, such a project would require not just a couple of namespaces – but a complete rewrite of the RTL. The assembler source-code and parser can be found in the “Next Generation Demos” folder (Smart Mobile Studio 3.0 demos). Feel free to build on the codebase if you fancy creating your own language;Get creative and enjoy! **Note: Using the assembler in your own projects requires a valid Smart Mobile license.

Native Apps

It’s interesting that people still ask this, since its one of our central advantages. We already generate native apps via the Phonegap post-processor.

phonegap

Phonegap turns your JS apps into native apps

Phonegap takes your compiled Smart (visual projects only) compiled code, processes it, and spits out native apps for every mobile OS on the market (and more). So you don’t have to compile especially for iOS, Android, Tizen or FireOS — Phonegap generates one for each system you need, ready for AppStore.

So we have native covered by proxy. And with millions of users Phonegap is not going anywhere.

Release date

We are going over the last beta as I type this, and Smart Mobile Studio 3.0 should be available next week. Which day is not easy to say, but at least before next weekend if all goes accoring to plan.

Make sure you visit www.smartmobilestudio.com and buy your license!

Patching Smart Mobile Studio’s ACE editor

June 30, 2018 Leave a comment

The Ace text-editor has been a part of the Smart Mobile Studio component set for a while now. It is seen by many as the de-facto code and text editor for JavaScript, and much like SynEdit for Delphi and C++ builder – Ace has support for a myriad of themes, languages and even key shortcut mapping.

Align problems

22814515_1630289797034370_9138255627706616601_nWith the introduction of our new theme engine, we completely revamped the entire notion of how a theme is organized. Gone are the hard-coded styles that targeted each individual control regardless if it was used or not. Instead we created a theme engine with a fixed number of borders and backgrounds, which are used as building-blocks by our visual controls.

This makes life much easier for everyone, especially Smart developers who write their own custom-controls (which you kinda have to do if you want something unique).

But Ace didn’t like this one bit. It has taken quite a debugging chore to track down what the heck is causing Ace to mis-place the cursor like that. It only happens when you apply a language theme to ace (Ace has its own themes and language parsers). And it’s not a superficial bug either, it renders Ace useless for anything serious.

Fixing the padding

The “bug” turned out to be as simple as padding. In our theme-files we are very careful and avoid imposing on other styles that might be loaded. But there are two sections where we apply values globally (as in “apply this to all elements of type x, y and z”).

One of these values is the padding. Depending on the theme, the padding is either set to 1px or 2px. This is set in a constant (Smart supports scriptable stylesheets) almost at the top of the file.

Before you start changing the theme files, I suggest you do the following:

  • Copy the existing theme files and prefix them with “np” (no padding). Just keep these copies in the same folder as the other themes
  • You should now have the following files in your theme folder:
    • npDefault.css
    • npAndroid.css
    • npiOS.css
    • Default.css
    • Android.css
    • iOS.pcss
  • Now edit each file (only those prefixed with np), and change the constant “stdpadding” which is defined on the top of each file (line #6), and set it to “0px” rather than the original “2px”.
  • Save all changes to the files
  • Restart Smart Mobile Studio

When the Smart IDE restarts it will have your additional theme files in the project options (under “linker”).

If you use Ace in your application then simply pick one of the new files as an alternative to the older. This fixes the problem with Ace’s cursor ending up behind the last character on a styled line.

Less intrusive fix

An alternative and less intrusive remedy, is to define a custom css style for Ace directly in your code. This is now very simple thanks to our css classes, but if you use Ace a lot then the above fix is probably the best for now.

stylecode

Injecting a CSS style is very simple 

Smart Pascal file enumeration under node.js

May 10, 2018 Leave a comment

Ok. I admit it. Writing an RTL from scratch has been one of the hardest tasks I have ever undertaken. Thankfully I have not been alone, but since I am the lead developer for the RTL, it naturally falls on me to keep track of the thousands of classes it comprises; how each affect the next, the many inheritance chains and subsequent causality timelines that each namespace represents.

We were the first company in the world to do this, to establish the compiler technology and then author a full RTL on top of that – designed to wrap and run on top of the JavaScript virtual machine. To be blunt, we didn’t have the luxury to looking at what others had done before us. For every challenge we have had to come up with solutions ourselves.

Be that as it may, after seven years we have gotten quite good at framework architecture. So whenever we need to deal with a new runtime environment such as node.js Рwe  have already built up a lot of experience with async JSVM development, so we are able to absorb and adapt much faster than our competitors.

Digging into a new platform

Whenever I learn a new language, I typically make a little list of “how do I do this?” type questions. It can be simple, like writing text to stdout, or more elaborate like memory mapped files, inheritance model, raw memory access and similar topics.

But one of the questions have always been: how do I enumerate files in a folder?

While this question is trivial at best, it stabs at the heart of the sub structure of any language. On operating systems like Linux a file is not just data on a disk like we are used to from Windows. A file can be a socket, a virtual access point exposed by the kernel, a domain link, a symbolic link or a stream. So my simple question is actually motivated to expose the depth of the language im learning. I then write down whatever topics come up and then research / experiment on them separately.

Node, like the browser, executes code asynchronously. This means that the code you write cannot be blocking (note: node does support synchronous file IO methods, but you really don’t want to use them in a server. They are typically used before the server is started to load preferences files and data).

As you can imagine, this throws conventional coding out the window. Node exposes a single function that returns an array of filenames (array of string), which helps, but it tells you nothing about the files. You don’t get the size, the type, create and modify timestamps – just the names.

To get the information I just mentioned you have to call a function called “fs.stat”. This is a common POSIX filesystem command. But again we face the fact that everything is async, so that “for / next” loop is useless.

Luke Filewalker

In version 3.0 of Smart Mobile Studio our Node.JS namespace (collection of units with code) has been upgraded and expanded considerably. We have thrown out almost all our older dependencies (like utf8.js and base64.js) and implemented these as proper codec classes in Smart Pascal directly.

Our websocket framework has been re-written from scratch. We threw out the now outdated websocket-io and instead use the standard “ws” framework that is the most popular and actively maintained module on NPM.

We have also implemented the same storage-device class that is available in the browser, so that you can write file-io code that works the same both server-side and client-side. The changes are in the hundreds so I wont iterate through them all here, they will be listed in detail on the release-notes document when the time comes.

But what is a server without a fast, reliable way of enumerating files?

Well, here at the Smart Company we use our own products. So when writing servers and node micro-services we face the exact same challenges as our customers would. Our job is to write ready solutions for these problems, so that you don’t have to spend days and weeks re-inventing the wheel.

Enumerating files is handled by the class TNJFileWalker (I was so tempted to call it Luke). This takes care of everything for you, all the nitty-gritty is neatly packed into a single, easy to use class.

Here is an example:

luke

Enumerating files has literally been reduced to childs play

The class also expose the events you would expect, including a filtering event where you can validate if a file should be included in the final result. You can even control the dispatching speed (or delay between item processing) which is helpful for payload balancing. If you have 100 active users all scanning their files at the same time -you probably want to give node the chance to breathe (20ms is a good value).

The interface for the class is equally elegant and easy to understand:

luke2

What would you prefer to maintain? 500.000 lines of JavaScript or 20.000 lines of pascal?

Compare that to some of the spaghetti JavaScript developers have to live with just to perform a file-walk and then do a recursive “delete folder”. Sure hope they check for “/” so they don’t kill the filesystem root by accident.

const fs = require('fs');
const path = require('path');

function filewalker(dir, done) {
    let results = [];

    fs.readdir(dir, function(err, list) {
        if (err) return done(err);

        var pending = list.length;

        if (!pending) return done(null, results);

        list.forEach(function(file){
            file = path.resolve(dir, file);

            fs.stat(file, function(err, stat){
                // If directory, execute a recursive call
                if (stat && stat.isDirectory()) {
                    // Add directory to array [comment if you need to remove the directories from the array]
                    results.push(file);

                    filewalker(file, function(err, res){
                        results = results.concat(res);
                        if (!--pending) done(null, results);
                    });
                } else {
                    results.push(file);

                    if (!--pending) done(null, results);
                }
            });
        });
    });
};

function deleteFile(dir, file) {
    return new Promise(function (resolve, reject) {
        var filePath = path.join(dir, file);
        fs.lstat(filePath, function (err, stats) {
            if (err) {
                return reject(err);
            }
            if (stats.isDirectory()) {
                resolve(deleteDirectory(filePath));
            } else {
                fs.unlink(filePath, function (err) {
                    if (err) {
                        return reject(err);
                    }
                    resolve();
                });
            }
        });
    });
};

function deleteDirectory(dir) {
    return new Promise(function (resolve, reject) {
        fs.access(dir, function (err) {
            if (err) {
                return reject(err);
            }
            fs.readdir(dir, function (err, files) {
                if (err) {
                    return reject(err);
                }
                Promise.all(files.map(function (file) {
                    return deleteFile(dir, file);
                })).then(function () {
                    fs.rmdir(dir, function (err) {
                        if (err) {
                            return reject(err);
                        }
                        resolve();
                    });
                }).catch(reject);
            });
        });
    });
};

Writing Smart Pascal Controls, async initialization and the tao pattern

May 7, 2018 Leave a comment

Async programming can take a bit getting used to if you come straight from Delphi or Lazarus. So in this little article I am going to show you an initialization pattern that will help you initialize your custom-controls and forms in way that is reliable.

Object Ready

In 99.9% of the custom-controls you create, you will either inherit directly from an existing control (like TW3Button, TW3EditBox or other traditional visual controls), or directly from TW3CustomControl.

If you have a quick look at the source for the RTL, which we take for granted that you do, you will find that our RTL is very familiar. It is loosely based on the LCL (lazarus component library), VCL (Visual component library) and with a dash of Mono GTK# thrown in for good measure. But while familiar in appearance, it really is a completely new RTL written to deliver the best of what HTML5 / JS has to offer.

One of the more interesting methods of TW3CustomControl is ObjectReady. This is actually introduced further down in the inheritance chain with TW3MovableControl, but most developers want the infrastructure TW3CustomControl delivers – so that will be the focus on the topic today.

In short, ObjectReady is called when your visual control has been created, injected into the DOM and is ready for use.

The common mistake

A common mistake with ObjectReady() is that the ready state somehow covers any child elements you might have created for your control. This is not the case. ObjectReady() is called when the current control is finished with its initialization, and is ready for manipulation.

Just before the ObjectReady() method is called, the csReady flag is added to the ComponentState set (note: if you don’t know what a set is, it’s a bit like an array of enums. Please google “Delphi sets” to investigate this further if you are just starting out).

Checking if a control is ready can be done manually by reading the csReady state from a controls ComponentState. But naturally, that only works if the control has already reached that state. Prior to the ready state the csCreating state is added to ComponentState, this is removed as the initialization completes and the control enters ready state.

The united states of custom-controls

To better understand when the different component states are set and what really happens when you create a visual control, let’s go through the steps one by one.

  • TW3TagObj
    • Ordinary constructor (create) is called
      • csCreating is added to ComponentState
      • DOM element name to be managed is obtained via TW3TagObj.MakeElementTagId()
    • Handle is obtained via TW3TagObj.MakeElementTagObj()
      • csLoading in added to ComponentState
      • A DOM level identifier (name) is assigned to the control
      • ZIndex is calculated and assigned to the control
    • StyleTagObject() method is called for any css adjustments
    • InitializeObject() is called, this is the constructor in our RTL
    • Control instance is registered with the global control tracker
      • csCreating is removed from ComponentState
      • csLoading in removed from ComponentState
  • TW3MovableControl
    • Alpha blending is initialized but not activated
    • if cfIgnoreReadyState is set in CreationFlags() then ObjectReady is called immediately without any delay
    • If not¬†cfIgnoreReadyState is set, the ReadySync() method is called

the ReadySync() method is of special importance here.

Since JavaScript is asynchronous, reality is that whatever child controls you have created during InitializeObject, can still be under construction even after InitializeObject() finishes. The JavaScript engine might have returned a handle, but the data for the instance is still under construction behind the scenes.

To be blunt: Never trust JavaScript to deliver a 100% ready to use element. If the browser is under heavy stress from other websites and internal work, that can have catastrophic consequences on the state of the objects it returns.

This is one of many reasons that made us chose to write our RTL from scratch rather than just fork CLX or try to be as Delphi friendly as possible. That would have been much easier for us, but it would also be to sell you on the tooth-fairy because that’s not how JavaScript works.

We want our users to have full control and enjoy the same freedom and simplicity that made us fall in love with object pascal all those years ago. And if we forced JavaScript into a pre-fabricated mold like the LCL; the spark and flamboyance that JavaScript brings to the table would have been irreparably damaged if not lost.

But let’s get back on topic.

Like mentioned above, ReSync() is of special importance. It will continuously check if the control is “ready” on a 10ms interval, and it keeps going until the criteria matches or it times out. To avoid infinite loops it has a maximum call stack of 300. Meaning it will keep trying 300 times, a total of 3 seconds and then break out free for safety reasons.

But once the criteria for ready-state matches (or the waiting interval times out)  РObjectReady() is finally called.

Keep your kids in order

While knowing when the control is ready for us is great for writing components, what practical purpose does it really serve if the child controls is excluded?

Well, again we are back at freedom. We could have baked in a wait sequence for our forms (since the designer knows what child elements are involved). But sadly that wont work on custom controls that the IDE doesn’t manage. And it would only work on forms.

A negative side-effect of this (yet I did test it) is that a form will remain blank until all child controls,¬†their children and their grand children – all reports “ready”.

In short: our code cannot manage what it doesn’t know. The IDE cannot (for reasons obvious) know what your code creates at runtime. And in large and complex controls like grids, planners or MDI systems – such code would get in your way and render the benefits null and void quickly.

As of writing there are some creative solutions to this, trying to get the timing right

  • Write your own checking routines inspired by ReadySync
  • Ignore the whole thing and just check ready-state and that child elements are not NIL in Resize(). This is what most people do.
  • Use TW3Dispatch and throw in an extra Resize() call somewhere in ObjectReady()

While perfectly legal (or perhaps better said: not illegal), these solutions are not very reliable. If the browser is under stress it can prioritize your layout code as less important – and suddenly you have a button where it’s not supposed to be, or a panel that never stretched as planned.

The Tao pattern

Tao (time aware operation) is pattern I created to solve this problem with a bit of grace. Much like the ReadySync() method we talked about earlier, it performs interval based checking of child element states, and thus allows you to do operations in a timely fashion.

As you probably know, under Smart Pascal you are not supposed to override the constructor when you create new controls. Instead you override InitializeObject(). The same goes for the destructor, there you override FinalizeObject().

So the 5 “must-know” methods for writing your own controls are:

  1. InitializeObject
  2. FinalizeObject
  3. ObjectReady
  4. Resize
  5. StyleTagObject

Note: Since Smart Mobile Studio has an evolved theme engine, it is rare that people override StyleTagObject() these days. But there are cases where you want to set some HTML attribute or alter a style; changes that are too small to justify a new style in the global stylesheet. It’s also the place to call ThemeReset() if you don’t want your control to use themes at all, or perhaps set a different theme border and background.

OK, let’s look at a practical example of how TAO works. It is simple, flexible and requires minimal adaptation if you want to adjust older controls you have made.

taoselect

Lets build a simple path selector control. Easy and ad-hoc

In this example we will be making a path selector. This is essentially an edit box with a button situated at the far-right. Clicking the button would bring up some form of dialog. I am excluding that for brevity since it’s not the control that is interesting here, but rather how we initialize the control.

type

  TTaoControl = class(TW3CustomControl)
  private
    FButton:  TW3Button;
    FEdit:    TW3EditBox;
  protected
    procedure InitializeObject; override;
    procedure FinalizeObject; override;
    procedure ObjectReady; override;
    procedure StyleTagObject; override;
    procedure Resize; override;
  end;

As you can see the control class is defined exactly the same way as before. There is no change what so ever in how you write your classes. Now let’s look at the implementation:

procedure TTaoControl.InitializeObject;
begin
  inherited;
  TransparentEvents := false;
  SimulateMouseEvents := false;

  // Create our editbox
  FEdit := TW3EditBox.Create(self);
  FEdit.SetSize(80, 28);

  // reate our select button
  FButton := TW3Button.Create(self);
  FButton.SetSize(70, 28);
end;

procedure TTaoControl.FinalizeObject;
begin
  FEdit.free;
  FButton.free;
  inherited;
end;

procedure TTaoControl.ObjectReady;
begin
  inherited;
  // set some constraints (optional)
  Constraints.MinWidth := 120;
  Constraints.MinHeight := 32;
  Constraints.Enabled := true;

  // TAO: Wait for the child controls to reach ready-state
  TW3Dispatch.WaitFor([FEdit, FButton], 5,
    procedure (Success: boolean)
    begin
      if Success then
      begin
        // set some properties for the edit box
        FEdit.ReadOnly := true;
        FEdit.PlaceHolder := 'Please selected a file';

        // set caption for button
        FButton.Caption := 'Select';

        // Do an immediate resize
        Resize();
      end;
    end);
end;

procedure TTaoControl.StyleTagObject;
begin
  inherited;

  // Set a container border. This border is
  // typically used by TW3Panel and TW3GroupBox
  ThemeBorder := btContainerBorder;
end;

procedure TTaoControl.Resize;
var
  LBounds:  TRect;
  dx, dy: integer;
  wd, EditWidth: integer;
begin
  inherited;
  // Make sure we dont do anything if resize has been
  // called while the control is being destroyed
  if not (csDestroying in ComponentState) then
  begin
    // Make sure we have ready state
    if (csReady in ComponentState) then
    begin
      // Check that child elements are all assigned
      // and that they have their csReady flag set in
      // ComponentState. This can be taxing. A more lightweight
      // version is TW3Dispatch.Assigned() that doesnt check
      // the ready state (see class declaration for more info)
      if TW3Dispatch.AssignedAndReady([FButton, FEdit]) then
      begin
        // Finally: layout the controls. This can be
        // optimized quite a bit, but focus is not on
        // layout code, but rather the sequence in which operations
        // are executed and handled.
        LBounds := ClientRect;
        wd := LBounds.width;
        dy := (LBounds.Height div 2) - (FEdit.Height div 2);
        EditWidth := (wd - FButton.Width) - 4;
        FEdit.SetBounds(LBounds.left, dy, EditWidth, FEdit.Height);

        dx := LBounds.left + EditWidth + 2;
        dy := (LBounds.Height div 2) - (FButton.Height div 2);
        FButton.SetBounds(dx, dy, FButton.Width, FButton.Height);
      end;
    end;
  end;
end;

If you look closely, what we do here is essentially to spread the payload and cost of creating child elements over multiple methods.

We reduce the constructor, InitializeObject(), to just creating our child controls and setting some initial sizes. This last point, setting initial size, is actually important. Because if the control has no size (width = 0, or height = 0) the browser will not treat the element as visible. Which in turn causes TW3Dispatch.WaitFor() to wait until a size is set.

TW3Dispatch methods

TW3Dispatch is a partial class. This is something that neither Delphi or Freepascal supports, and it has it’s root in C# and the dot net framework.

In short it means that a class can have it’s implementation spread over multiple files. So instead of having to complete a class in a single unit, or inherit from a class and then expanding it – partial classes allows you to expand a class over many units.

This is actually really cool and useful, especially when you work with multiple targets. For example, TW3Dispatch is first defined in System.Time.pas which is the universal namespace that only contains code that runs everywhere. This gives you the basic functionality like delayed execution (just open the unit and have a look).

The class is then further expanded in SmartCL.Time (SmartCL being the namespace for visual, HTML based JavaScript applications). There it gains methods like RequestAnimationFrame() which doesnt exist under node.js for example.

Smart Mobile Studio’s namespaces makes good use of partial classes

TW3Dispatch is further expanded in SmartCL.Components.pas, which is the core unit for visual controls. So starting with version 3.0 the functions I have just demonstrated will be available in the RTL itself.

Until then, you can download TW3Dispatch with the TAO methods here. You need to put it in your own unit, and naturally – use it with care.

Click here to download the TW3Dispatch code.

Note: This code is not free or open-source. It is indended for Smart Mobile Studio owners exclusively, and made available here so registered users can start working with the control coding pattern.

Using Smart Mobile Studio under Linux

April 22, 2018 Leave a comment

Every now and then when I post something about Smart Mobile Studio, an individual or two wants to inform me how they cannot use Smart because it’s not available for Linux. While more rare, the same experience happens now and then with OS X.

linux

The Smart desktop demo running in Firefox on Ubuntu, with Quake 3 at 60 fps

While the request for Linux or OS X support is both valid and understandable (and something we take seriously), more often than not these questions can be a pre-cursor to a larger picture; one that touches on open-source, pricing and personal philosophical points of view; often with remarks on pricing.

Truth be told, the price we ask for Smart Mobile Studio is close to symbolic. Especially if you take the time to look at the body of work Smart delivers. We are talking hundreds of hand written units with thousands of classes, each spesifically adapted for HTML5, Phonegap and Node.js. Not to mention ports of popular JavaScript frameworks.

If you compare this to native object pascal development tools with similar functionality, they can set you back thousands of dollars. Our asking price of $149 for the pro edition, $399 for the enterprise edition, and a symbolic $42 for the basic edition, that is an affordable solution. Also keep in mind that this gives you access to updates for a duration of 12 months. When was the last time you bought a full development suite that allows you to write mobile applications, platform independent servers, platform independent system services and HTML5 web applications for less that $400 ?

prices

Our price model is more than reasonable considering what you get

With platform independent we mean that in the true sense of the word: once compiled, no changes are required. You can write a system service on Windows and it will run just fine under Linux or OS X. No re-compile needed. You can take a server and copy it to Amazon or Azure, run it in a cluster or scale it from a single instance to 100 instances without any change. That is unheard of for object pascal until now.

Smart Mobile Studio is the only object pascal development system that delivers a stand-alone IDE, stand-alone compiler, a wast object-oriented run-time library written from scratch to cover HTML5, Node.js and embedded systems that run JavaScript.

And yes, we know there are other systems in circulation, but none of them are even close to the functionality that we deliver. Functionality that has been polished for seven years now. And our RTL is growing every day to capture and expose more and more advanced functionality that you can use to enrich your products.

21272810_1585822718147745_4358597225084661216_o

The RTL class browser shows the depth of our RTL

As of writing we have a team of six people working on Smart Mobile Studio. We have things in our labs that is going to change the way people build applications forever. We were the first to venture into this new landscape. There were nobody we could imitate, draw inspiration from or learn from. We had to literally make the path as we moved forward.

And our vision and goal remains the same today as it was seven years ago: to empower object pascal developers – and to secure their investment in the language and methodology that is object pascal.

Discipline and purpose

There is so much I would like to work on right now. Things I want to add to Smart Mobile Studio because I find them interesting, powerful and I know people are going to love them. But that style of development, the “Garage days” as people call it, is over. It does wonders in the beginning of a project maybe, but eventually you reach a stage where a formal timeline and business plan must be carved in stone.

And once defined, you have to stick to it. It would be an insult to our customers if we pivoted left and right on a whim. Like every company we have room for research, even a couple of skunkwork projects, but our primary focus is to make our foundation rock solid before further growth.

j5

By tapping into established JavaScript frameworks you can cover more than 40+ embedded systems and micro-controllers. More and more hardware supports JS for automation

Our “garage days” ended around three years ago, and through hard work we defined our timeline, business model and investor program. In 2017 we secured enough capital to continue full-time development.

Our timeline has been published earlier, but we can re-visit some core points here:

The visual components that shipped with Smart Mobile Studio in the beginning, were meant more as examples than actual ready-to-use modules. This was common for other development platforms of the day, such as Xamarin’s C# / Mono toolchain, where you were expected to inherit from and implement aspects of a “partial class”. This is also why Smart Pascal has support for partial classes (neither Delphi or Freepascal supports this wonderful feature).

native

One of our skunkwork projects is a custom linux distro that runs your Smart applications directly in the Linux framebuffer. No X or desktop, just your code. Here running “the smart desktop” as the only visual front-end under x86 vmware

Since developers coming from Delphi had different expectations, there was only one thing to do. To completely re-write every single visual control (and add a few new controls) so that they matched our customers expectations. So the first stretch of our timeline has been 100% dedicated to the visual aspects of our RTL. While doing so we have made the RTL faster, more efficient, and added some powerful sub-systems:

  • A dedicated theme engine
  • Unified event delegation
  • Storage device classes
  • Focus and control tracking
  • Support for relative position modes
  • Support for all boxing models
  • Inline linking ( {$R “file.js”} will now statically link an external library)
  • And much, much more

So the past eight months has been all about visual components.

22814515_1630289797034370_9138255627706616601_n

Theming is important

The second stretch, which we are in right now, is dedicated to the non-visual infrastructure. This means in particular Node.js but also touches on non-visual components, TAction support and things that will appear in updates this year.

Node.js is especially important since it allows you to write platform and chipset independent web servers, system services and command-line applications. This is pioneering work and we are the first to take this road. We have successfully tamed the alien landscape of JavaScript, both for client, mobile and server Рand terraformed it into a familiar, safe and productive environment for object pascal developers.

I feel the results speak for themselves, and our next update brings Smart Mobile Studio to the next level: full stack cloud and web development. We now cover back-end, middle-ware and front-end technologies. And our RTL now stretches from micro-controllers to mobile application to clustered cloud services.

This is the same technology used by Netflix to process terabytes of data every second on a global scale. Which should tell you something about the potential involved.

Working on Linux

Since Smart Mobile Studio was designed to be a swiss army knife for Delphi and Lazarus developers, capable to reaching segments of the market where native code is unsuitable – our primary focus is Microsoft Windows. At least for now.

Delphi itself is a Windows-based development system, and even though it supports multiple targets, Windows is still the bread and butter of commercial software development.

Like I mentioned above, we have a timeline to follow, and until we have reached the end of that line, we are not prepared to refactor our IDE for Linux or OS X. This might change sooner than people think, but until our timeline for the RTL is concluded, we will not allocate time for making the IDE platform independent.

But, you can actually run Smart Mobile Studio on both Linux and OS X today.

Linux has a system called Wine. This is a system that implements the Windows API, but delegates all the calls to Linux. So when a Windows program calls a WinAPI through Wine, its delegated to Linux variation of the same call. This is a massive undertaking, but it has years of work behind it and functions extremely well.

So on linux you can install it by opening up a shell and typing:

sudo apt-get install wine

I take for granted here that your Linux flavour has aperture installed (I’m using Ubuntu since that is easy to work with), which is the package manager that gives you the “apt-get” command. If you have some other system then just google how to install a package.

With Wine and it’s dependencies installed, you can run the Smart Mobile Studio installer. Wine will create a virtual, sandboxed disk for you – so that all the files end up where they should. Once finished you punch in the license serial number as normal, and voila – you can now use Smart Mobile Studio on Linux.

Note: in some cases you have to right-click the SmartMS.exe and select “run with -> Wine”, but usually you can just double-click the exe file and it runs.

Smart Mobile Studio on OSX

Wine has been ported to OS X, but it’s more user friendly. You download a program called wine-bottler, which takes Smart and bundles it with wine + any dependencies it needs. You can then start Smart Mobile Studio like it was a normal OS X application.

Themes and look

The only problem with Wine is that it doesnt support Windows themes out of the box. It would be illegal for them to ship those files. But you can manually copy over the windows theme files and install them via the Wine config application. Once installed, Smart will look as it should.

By default the old Windows 95 look & feel is used by Wine. Personally I dont care too much about this, it’s being able to code, compile and run the applications that matters to me – but if you want a more modern look then just copy over the Windows theme files and you are all set.

 

 

Multiplication decomposer

April 22, 2018 Leave a comment

Here is a fun little number decomposer I made a while back. As you might know, the ancient egyptians were no strangers to binary. They also didnt use multiplication tables like we do – but instead used a method called “double down”.

To demonstrate this technique I wrote a simple little program that takes a multiplication and then breaks it down to the numbers an egyptian would look-up in his table to find the right answere.

egypt

It’s actually pretty cool. They did not apply multiplication like we do at all, but adding

You will need to drop two textboxes, 3 labels, one button and a memo control on your form (see layout in picture).

 

unit Form1;

interface

uses
  System.Types,
  System.Types.Convert,
  System.Objects,
  System.Time,
  SmartCL.System,
  SmartCL.Time,
  SmartCL.Graphics,
  SmartCL.Components,
  SmartCL.FileUtils,
  SmartCL.Forms,
  SmartCL.Fonts,
  SmartCL.Theme,
  SmartCL.Borders,
  SmartCL.Application,
  SmartCL.Controls.Button,
  SmartCL.Controls.Label,
  SmartCL.Controls.EditBox,
  SmartCL.Controls.Memo;

type

  TDecomposer = Class(TObject)
  private
    FBitValues:   TIntArray;
    FLeftPillar:  TIntArray;
    FRightPillar: TIntArray;
    FSumPillar:   TIntArray;
  protected
    procedure BuildLeftPillar(Number:Integer);
    procedure BuildRightPillar(Multiplier:Integer);
  public
    Property  LeftPillar:TIntArray read FLeftPillar;
    Property  RightPillar:TIntArray read FRightPillar;
    Property  SumPillar:TIntArray read FSumPillar;

    function  DecomposeNumber(aNumber:Integer):String;
    function  DecomposeMultiplication(aNumber,aMultiplier:Integer):String;

    Constructor Create;Virtual;
  end;

  TForm1 = class(TW3Form)
    procedure W3Button1Click(Sender: TObject);
  private
    {$I 'Form1:intf'}
  protected
    procedure InitializeForm; override;
    procedure InitializeObject; override;
    procedure Resize; override;
  end;

implementation

{ TForm1 }

procedure TForm1.W3Button1Click(Sender: TObject);
begin
  var LObj := TDecomposer.Create;
  try
    var LText := LObj.DecomposeMultiplication(StrToInt(w3editbox1.Text),StrToInt(w3editbox2.text));
    w3memo1.text := LText;
    writeln(LText);
  finally
    LObj.free;
  end;
end;

procedure TForm1.InitializeForm;
begin
  inherited;
  // this is a good place to initialize components
end;

procedure TForm1.InitializeObject;
begin
  inherited;
  {$I 'Form1:impl'}
end;

procedure TForm1.Resize;
begin
  inherited;
end;

//#############################################################################
// TDecomposer
//#############################################################################

constructor TDecomposer.Create;
begin
  inherited Create;

  (* Build table of bitvalues *)

  var mValue := 1;
  for var x :=1 to 32 do
  begin
    FBitValues.add(mValue);
    mValue:=mValue * 2;
  end;
end;

procedure TDecomposer.BuildLeftPillar(Number:Integer);
begin
  FLeftPillar.clear;
  if FBitValues.length>0 then
  begin
    var mValue := 1;
    for var x := FBitValues.low to FBitValues.high do
    begin
      if FBitValues[x] <= Number then         FLeftPillar.add(FBitValues[x])       else         break;     end;   end; end; procedure TDecomposer.BuildRightPillar(Multiplier:Integer); begin   FRightPillar.clear;   if FLeftPillar.length>0 then
  begin
    for var x := FLeftPillar.low to FLeftPillar.high do
    begin
      FRightPillar.add(Multiplier);
      Multiplier:=Multiplier * 2;
    end;
  end;
end;

function TDecomposer.DecomposeMultiplication
         (aNumber,aMultiplier:Integer):String;
begin
  var mSum := aNumber * aMultiplier;
  BuildRightPillar(aMultiplier);

  result := aNumber.toString + ' x '
     + aMultiplier.toString
     + ' = '
     + DecomposeNumber(mSum);
end;

function TDecomposer.DecomposeNumber(aNumber:Integer):String;
begin
  FSumpillar.clear;
  FLeftPillar.clear;
  FRightPillar.clear;

  BuildLeftPillar(aNumber);

  for var x := FLeftPillar.low to FLeftPillar.High do
  begin
    if TInteger.getBit(x,aNumber) then
    FSumPillar.add(FBitValues[x]);
  end;

  if FSumPillar.length>0 then
  Begin
    result:=aNumber.ToString + ' = ';
    for var x:=FSumPillar.low to FSumpillar.high do
    begin
      if x=FSumPillar.low then
      result += FSumPillar[x].toString else
      result += ' + ' + FSumPillar[x].toString;
    end;
  end;
end;

function QTX_GetNumberProducer(aNumber:Integer):String;
const
  QTX_BITS:  Array[0..31] of Integer =
  ( 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048,
    4096, 8192, 16384, 32768, 65536, 131072, 262144, 524288,
    1048576, 2097152, 4194304, 8388608, 16777216, 33554432,
    67108864, 134217728, 268435456, 536870912, 1073741824,
    2147483648);
Begin
  var LStack: array of string;
  if aNumber > 0 then
  begin
    for var x := QTX_BITS.low to QTX_BITS.high do
    begin
      if TInteger.getBit(x,aNumber) then
        LStack.add(QTX_BITS[x].toString);
      if QTX_BITS[x] >= aNumber then
        break;
    end;

    if LStack.length>0 then
    begin
      result := aNumber.toString + ' = ';
      for var x := LStack.low to LStack.high do
      begin
        if x > LStack.low then
          result += ' + ' + LStack[x]
        else
          result += LStack[x];
      end;
    end;
  end else
    result := "0";
end;

function QTX_GetNumberMultiplier(aFirst,aSecond:Integer):String;
Begin
  var LSum := aFirst * aSecond;
  result := aFirst.ToString() + ' x ' + aSecond.ToString();
  result += ' = ';
  result += QTX_GetNumberProducer(LSum);
end;

initialization
  Forms.RegisterForm({$I %FILE%}, TForm1);
end.

Smart Mobile Studio 3.0 and beyond

March 20, 2018 Leave a comment

cascade_03With Smart Mobile Studio 3.0 entering its second beta, Smart Pascal developers are set for a boost in quality, creativity and power. We have worked extremely hard on the product this past year, including a complete rewrite of all our visual controls (and I mean all). We also introduced a completely new theme engine, one that completely de-couples visual appearance from structural architecture (it also allows scripting inside the CSS theme files).

All of that could be enough for a version bump, but we didn’t stop there. Much of the sub-strata in Smart has been¬†re-implemented. Focus has been on stability, speed and future growth. The system is now divided into a set of name-spaces (System, SmartCL, SmartNJ, Phonegap, and Espruino), making it easier to navigate between the units as well as expanding the codebase in the future.

To better understand the namespaces and why this is a good idea, let’s go through how our units are organized.

smart_namespace

The RTL is made to expand easily and preserve as much functionality as possible

  • The System namespace is the foundation. It contains clean, platform independent code. Meaning code that doesn’t rely on the DOM (browser) or runtime (node). Focus here is on universal code, and to establish common object-pascal classes.
  • Our SmartCL namespace contains visual code, meaning code and controls that targets the browser and the DOM. SmartCL rests on the System namespace and draws functionality from it. Through partial classes we also expand classes introduced in the system namespace. A good example is System.Time.pas and SmartCL.Time.pas. The latter expands the class TW3Dispatch with functionality that will only work in the DOM.
  • SmartNJ is our high-level nodejs namespace. Here you find classes with fairly complex behavior such as servers, memory buffers, processes and auxillary classes. SmartNJ draws from the system namespace just like SmartCL. This was done to avoid multiple implementations of streams, utility classes and common functions. Being able to enjoy the same functionality under all platforms is a very powerful thing.
  • Phonegap is our namespace for mobile devices. A mobile application is basically a normal visual application using SmartCL, but where you access extra functionality through phonegap. Things like access to a device’s photos, filesystem, dialogs and so on is all delegated via phonegap.
  • Espruino is a namespace for working with Espruino micro-controllers. This has been a very low-level affair so far, due to size limitation on these devices. But with our recent changes you can now, when you need to, tap into the system namespace for more demanding behavior.

As you can see there is a lot of cool stuff in Smart Mobile Studio, and our codebase is maturing nicely. With out new organization we are able to expand both horizontally and vertically without turning the codebase into a gigantic mess (the VCL being a prime example of how not to implement a multi-platform framework).

Common behavior

One of the coolest things we have added has to be the new storage device classes. As you probably know the browser has a somewhat “limited” storage mechanism. You are stuck with name-value pairs in the cache, or a filesystem that is profoundly frustrating to work with. To remedy this we took the time to implement a virtual filesystem (in memory filesystem) that emits data to the cache; we also implemented a virtual storage device stack on top of it, one for each target (!).

In short, if a target has IO capability, we have implemented a storage “driver” for it. So instead of you having to write 4-5 different storage mechanisms – you can now write the storage code once, and it works everywhere.

This is a pretty cool system because it doesn’t limit us to local device storage. We can have device classes that talk to Google-Storage, One-Drive, Dropbox and so on. It also opens up for custom storage solutions should you already have this pre-made on your server.

Database support, a quick overview

Databases have always been available in Smart Mobile Studio. We have units for WebSQL, IndexDB and SQLite. In fact, we even compiled SQLite3 from native C code to asm.js, meaning that the whole database engine is now pure JavaScript and no-longer dependant on W3C standards.

smart_db

Each DB engine is implemented according to a framework

Besides these we also have TW3Dataset which is a clean, Smart Pascal implementation of a single table dataset (somewhat inspired by Delphi’s TClientDataset). In our previous beta we upgraded TW3Dataset with a robust expression parser, meaning that you can now set filters just like Delphi does. And its all written in Smart Mobile Studio which means there are no dependencies.

 

And ofcourse, there is also direct connections to Embarcadero Datasnap servers, and Remobjects SDK servers. This is excellent if you have an already existing Delphi infrastructure.

A unified DB framework

If you were hoping for a universal DB framework in beta-2 of v3.0, sadly that will not be the case. The good news is that databases should make it into v3.2 at the latest.

Databases looks simple: table, rows and columns right? But since each database engine known to JavaScript is written different from the next, our model has to take height for these and be dynamic enough to deal with them.

The model we used with WebSQL is turning out to be the best way forward I feel, but its important to leave room for reflection and improvements.

So getting our DB framework established is a priority for us, and we have placed it on our timeline for (at the latest) v3.2. But im hoping to have it done by v3.1. So it’s a little ahead of us, but we need that time to properly evolve the framework.

Smart Desktop [a.k.a Amibian.js]

The feedback we have received on our Smart Desktop demos have been pretty overwhelming. It is also nice to know that our prototype is being used to deliver software to schools and educational centers. So our desktop is not going away!

smart_desktop

Fancy a game of Quake at 60+ fps? Web assembly rocks!

But we are not rushing into this without some thought first. The desktop will become a project type like I have written about many times before. So you will be able to create both the desktop and client applications for it. The desktop is suitable for software that requires a windowing environment (a bit like Sencha or similar frameworks). It is also brilliant for kiosk displays and as a remote application hub.

Our new storage device system came largely from Amibian, and with these now a part of our RTL we can clean up the prototype considerably!

Smart assembler

It may sound like an oxymoron, but a lab project we created while testing our parser framework (system.text.parser unit) turned into an exercise in compiler / assembler making. We implemented a virtual machine that runs instructions represented by bytecodes (fairly straight ahead stuff). It supports the most common assembler methods, vaguely inspired by the Motorolla 68k processor with a good dose of ARM thrown in for good measure.

smart_assembler

Yes that is a full parser, assembler and runtime model

If you ponder why on earth this would be interesting, consider the following: most web platforms allow for scripting by third-party developers. And by opening up for that these, the websites themselves become prone to attacks and security breaches. There is no denying that any JS based framework is very fragile when potentially hundreds of unknown developers are hacking away at it.

But what if you could offer third parties to write plugins using more traditional languages? Perhaps a dialect of pascal, a subset of basic or perhaps C#? Wouldnt that be much better? A language and (more importantly) runtime that you have 100% control over.

While our assembler, disassembler and runtime is still in its infancy (and meant as a demo and excercise), it has future potential. We also made the instructions in such a way that JIT compiling large chunks of it is possible – and the output (or codegen) can be replaced by for example web assembly.

Right now it’s just a curiosity that people can play with. But when we have more time I will implement high-level parsers and codegens that emit code via this assembler. Suddenly we have a language that runs under node.js, in the browser or any modern JS runtime engine – and its all done using nothing but Smart Mobile Studio.

Well, stay tuned for more!

Alternative pointers in Smart Mobile Studio

February 27, 2018 Leave a comment

Smart Mobile Studio already enjoy a rich and powerful set of memory handling classes and methods. If you have a quick look in the memory units (see below) you will find that Smart Mobile Studio really makes JavaScript sing and dance like no other.

As of writing in version 3.0 BETA the following units are dedicated to raw memory manipulation:

  • System.Memory
  • System.Memory.Allocation
  • System.Memory.Buffer
  • System.Memory.Views

Besides these, the unit System.Types.Convert represents the missing link. It contains the class TDataType which converts data between intrinsic (language level) data types and byte arrays.

Alternative pointers

While Smart has probably one of the best frameworks (if not THE best) for memory handling out there, including the standard library that ships with Node.js, the way it works is slightly different from Delphi’s and¬†Freepascal’s approach.

Since JavaScript is reference based rather than pointer based, a marshaling offset mechanism is more efficient in terms of performance; So we modeled this aspect of Smart on how C# in particular organized its memory stuff.

But, is it possible to implement more Delphi like pointers? To some degree yes. The best would be to do this on compiler level, but even without such deep changes to the system you can actually implement a more Delphi-ish interface.

Here is an example of just such a system. It is small and efficient, but compared to the memory units in the RTL it’s much slower. This is also why we abandoned this way of handling memory in the first place. But perhaps someone will find it interesting, or it can help you port over code from Delphi to HTML5.

unit altpointers;

interface

uses
  W3C.TypedArray,
  System.Types,
  System.Types.Convert,
  System.Memory,
  system.memory.allocation,
  System.Memory.Buffer,
  System.Memory.Views;

type

  Pointer = variant;

  TPointerData = record
    Offset: integer;
    Buffer: JArrayBuffer;
    View:   JUint8Array;
  end;

function IncPointer(Src: Pointer; AddValue: integer): Pointer;
function DecPointer(Src: Pointer; DecValue: integer): Pointer;
function EquPointer(src, dst : Pointer): boolean;

// a := a + bytes
operator + (Pointer,   integer): Pointer uses IncPointer;

// a := a - bytes
operator - (Pointer,   integer): Pointer uses DecPointer;

// if a = b then
operator = (Pointer,   Pointer): boolean uses EquPointer;

function  Allocmem(const Size: integer): Pointer;
function  Addr(const Source: Pointer; const Offset: integer): Pointer;
procedure FreeMem(const Source: Pointer);
procedure MemSet(const Target: pointer; const Value: byte); overload;
procedure MemSet(const Target: pointer; const Values: array of byte); overload;
function  MemGet(const Source: pointer): byte; overload;
function  MemGet(const Source: pointer; ReadLength: integer): TByteArray; overload;

implementation

function MemGet(const Source: pointer): byte;
begin
  if (Source) then
  begin
    var SrcData: TPointerData;
    asm @SrcData = @Source; end;
    result := SrcData.View.items[SrcData.Offset];
  end else
  raise Exception.Create('MemGet failed, invalid pointer error');
end;

function MemGet(const Source: pointer; ReadLength: integer): TByteArray;
begin
  if (Source) then
  begin
    var SrcData: TPointerData;
    asm @SrcData = @Source; end;

    var Offset := SrcData.Offset;

    while ReadLength > 0 do
    begin
      result.add( SrcData.View.items[Offset] );
      inc(Offset);
      dec(ReadLength);

      if offset >= SrcData.View.byteLength then
        raise Exception.Create('MemGet failed, offset exceeds memory');
    end;
  end else
  raise Exception.Create('MemGet failed, invalid pointer error');
end;

procedure MemSet(const Target: pointer; const Value: byte);
begin
  var DstData: TPointerData;
  asm @DstData = @Target; end;
  dstData.View.items[DstData.Offset] := value;
end;

procedure MemSet(const Target: pointer; const Values: array of byte);
begin
  if Values.length > 0 then
  begin
    var DstData: TPointerData;
    asm @DstData = @Target; end;

    var offset := DstData.Offset;
    for var x := low(Values) to high(Values) do
    begin
      dstData.View.items[offset] := Values[x];
      inc(offset);
      if offset >= DstData.View.byteLength then
        raise Exception.Create('MemSet failed, offset exceeds memory');
    end;
  end;
end;

function EquPointer(src, dst : Pointer): boolean;
begin
  if (src) then
  begin
    if (dst) then
    begin
      var SrcData: TPointerData;
      var DstData: TPointerData;
      asm @SrcData = @Src; end;
      asm @DstData = @dst; end;
      result := SrcData.buffer = dstData.buffer;
    end;
  end;
end;

function IncPointer(Src: Pointer; AddValue: integer): Pointer;
begin
  if (Src) then
  begin
    // Check that there is an actual change.
    // If not, just return the same pointer
    if AddValue > 0 then
    begin
      // Map source data
      var SrcData: TPointerData;
      asm @SrcData = @Src; end;

      // Calculate new offset, using the current view
      // position as the present location.
      var NewOffset := srcData.Offset;
      inc(NewOffset, AddValue);

      // Make sure the new offset is within the range of the
      // memory buffer. Picky yes, but this is not native land
      if  (NewOffset >=0)
      and (NewOffset  0 then
    begin
      // Map source data
      var SrcData: TPointerData;
      asm @SrcData = @Src; end;

      // Calculate new offset, using the current view
      // position as the present location.
      var NewOffset := srcData.Offset;
      dec(NewOffset, DecValue);

      // Make sure the new offset is within the range of the
      // memory buffer. Picky yes, but this is not native land
      if  (NewOffset >=0)
      and (NewOffset  0 then
  begin
    var Data: TPointerData;
    Data.Offset := 0;
    Data.Buffer := JArrayBuffer.Create(Size);
    Data.View := JUint8Array.Create(Data.Buffer, 0, Size);
    asm
      @result = @data;
    end;
  end else
  raise Exception.Create('Allocmem failed, invalid size error');
end;

function Addr(const Source: Pointer; const Offset: integer): Pointer;
begin
  if (Source) then
  begin
    if offset > 0 then
    begin
      // Map source data
      var SrcData: TPointerData;
      asm @SrcData = @Source; end;

      // Check that offset is valid
      if (Offset >=0) and (offset < srcData.buffer.byteLength) then
      begin
        // Setup new Pointer data
        var Data: TPointerData;
        Data.Buffer := SrcData.Buffer;
        Data.View := SrcData.View;
        Data.Offset := Offset;
        asm
          @result = @data;
        end;
      end else
      raise Exception.Create('Addr failed, offset exceeds memory');
    end else
    raise Exception.Create('Addr failed, invalid offset error');
  end else
  raise Exception.Create('Addr failed, invalid pointer error');
end;

procedure FreeMem(const Source: Pointer);
begin
  if (source) then
  begin
    // Map source data
    var SrcData: TPointerData;
    asm @SrcData = @Source; end;

    // Flush reference and let the GC take care of it
    SrcData.Buffer := nil;
    SrcData.View := nil;
    SrcData.Offset := 0;
    asm
      srcData = {}
    end;
  end else
  raise Exception.Create('FreeMem failed, invalid pointer error');
end;

end.

Using the pointers

As you can probably see from the code there is no such thing as PByte, PWord or PLongword here. We use a clean uint8 typed array that we link to a memory buffer, so “pointer” here is fully byte based despite it’s untyped origins. In reality it just holds a TPointerData structure, but since this is done via asm sections, the compiler cant see it and treats it as a variant.

The operators add support for code like:

var buffer := allocmem(1024);
memset(buffer, $ff);
buffer := buffer + 1;
memset(buffer, $FA)

But using the overloaded memset procedure is a bit more efficient:

var buffer := allocmem(1024);
var bytes := TDataType.StringToBytes('this is awesome!');
memset(buffer, bytes);
buffer := buffer + bytes.length;
// write more data here

While fun to play with and perhaps useful in porting over older code, I highly recommend that you familiarize yourself with classes like TBinaryData that represents a fully managed buffer with a rich number of methods to use.

And ofcourse let us not forget TMemoryStream combined with TStreamWriter and TStreamReader. These will no doubt feel more at home both under HTML5 and Node.js

Note: WordPress formatting of pascal code is not the best. Click here to view the code as PDF.