Moving to Linux, Part 3, Virtualization

I was going to look at backup services and how to build your own NAS and connect to it with freepascal (and more practical aspects of Linux) in this post, but due to the confusion regarding VMWare and how I work, I decided to dedicate this article to exactly that topic. apparently there are quite a few programmers out there who have little or no experience with VMWare or emulation at large – who quite frankly did not understand what I meant.

I do all my work in VMWare machines

First, to clear up any misunderstanding regarding “saving” money by moving to Linux. I was asked about this on my Delphi Developers Facebook group (if you havent joined it, then check it out – its probably the most active place online for Delphi developers). Well, it goes like this:

If you want to follow the law then you actually need A license for each operative system you run inside your virtual machines. So if you have Windows XP installed inside a virtual machine, that means you must buy a Windows XP license. The same goes for Windows Vista, Windows 7 and naturally Windows 8. From what I know you can clone as many instances of these as you wish, but you must own a license for each operative system.

Presently I have the following operative systems:

• Windows XP
• Windows Vista
• Windows 7
• Windows 8

That is in sub total 4 licenses that I have bought and own, meaning that I have the right to install them anywhere I wish.

Cloning

To clone a virtual machine means that you create a duplicate of a the installation disk using the same virtual hardware. When you run such a clone, the running version is called “an instance” in the world of virtualization.

A clone is a duplicate of a machine

So why do we clone virtual machines? Well it bakes down to configurations. You start by installing Windows into a virtual machine, at this point it’s completely fresh and without any software at all. Think of it as a base-class. I tend to clone from that point and then add software into the clone. Why? Because if something goes wrong with the clone, I can simply delete it and go back to the fresh. clean installation.

Snapshots

Another aspect of virtual machines is something called a “snapshot”. A snapshot simply makes a clone of your virtual machine state (as opposed to the whole shabam) as it is right now, and should you install more software into a snapshot or state – the virtual machine and other snapshots remains unaffected. You can later revert to that snapshot. The other machine state with the error can be deleted with a single click.

Snapshot delta disks

It’s actually the exact same thing as a Delphi class actually. You create X instances of a Delphi class, and the object you get is called (drumroll) an instance. So it’s the same old thing but on a grander scale.

Snapshots are extremely flexible and effective. You can create as many snapshots as you like, building up a whole tree-view of different machine states. It’s quite common to make snapshots before you install anything at all. So a lot of developers have 10-20 snapshots of their developer machine, allowing them to revert back to a previous state at any given time. Also, you can go back to a previous snapshot, almost like time travel, without anything happening to the state you left the machine. And should you add changes to a previous snapshot – that is isolated there and only there. Like a branch on a tree.

The snapshot layout in VMWare is easy to use

It is, for instance, common to Install a version of Delphi, then revert back to the root snapshot (remember, the Delphi snapshot is not deleted), then install another version of Delphi, and so on — In effect, your virtual machine will contain every Delphi version you own, isolated in their own reality and with absolutely no knowledge of each other. You simply jump to the snapshot with the version you need to work on and that’s it.

Virtualization is not just clever, it’s bloody awesome!

Cloud services

A lot of developers still havent got the message about cloud computing. The fact is, you don’t just upload html files and rent a database anymore — you upload a complete developer environment (!). Yes you read right, you upload a complete virtual machine, software and all, including whatever applications you want to run (server, webpage written in intraweb, mysql — install whatever you want to run in the cloud) to a cloud computer. And the cloud service provider will run X number of instances on their supercomputers for you. If you need to scale and have 10 instances active at the same time, that can be done in the admin console with a single click (watch you visa card here, you may burn yourself really bad if you do it wrong).

It’s not your daddy’s Windows anymore

So if you got the cash to do so, you can actually upload your VMWare clones and execute them in the cloud. You would log into your Windows instances using a normal terminal-server interface, and have your entire Delphi programming career online 24/7.

It may take some time to fully grasp the potential and delicacies of this.

Programming on VMWare

The VMWare desktop program (as opposed to the server, cloud based engine) is a piece of magic. Like mentioned in the beginning of this post, it allows you to create as many virtual machines as you like, and then install whatever operative system you desire into them. You can also clone machines, either in full or extended clones (only differences are stored) – and you can also do snapshots.

Do your work in a virtual machine

As you can probably imagine, moving all your Delphi installations into virtual machines opens up a whole world of benefits. First of all you dont have to carry a huge machine with you – because naturally you keep your virtual machines on en external USB drive (remember to back this up often!). All you really need is the virtual machine “player”, the VMWare program that executes your virtual machines, allowing you to do your work.

Testing also becomes easier with VMWare, because even if you don’t have a Windows XP box, you can still use it via a virtual machine. You just take a snapshot, install your software, test it, then revert back to the snapshot. Voila, whatever changes you made to the virtual machine is deleted and gone! And Windows XP is pristine clean like ever before.

Saving money

This was a topic people asked about quite a lot. How could I propose to save money by moving to Linux. Well that’s quite easy! In the beginning of this post I explained that you need to buy and own a license of each version of Windows you create virtual machines from. That is fair to do since Microsoft is the company making our programming lives possible – and i think it’s fair to actually buy the software you use. And by law you actually have to own Windows to use it, so there is no point running cracks in my view.

But what about the real computers that we run our virtual machines on? In my case I have a Macbook pro laptop and also a modern iMac. I also own a ton of PC’s. What about the license for Windows to dual-boot on my Mac’s?

So when I sum up the licenses I need I get the following:

• 4 virtual machine root installations of Windows
• 2 physical machine installations of Windows 7

Since all I really need to do my work in Windows are the virtual machines and the VMWare player, I can actually get rid of having to buy the two licenses for my “real” computers – and install Linux instead, thus saving the price of two Windows 7 licenses (ca. 1500 NKR a piece), saving in total $466.

As long as the VMWare player exists on a platform, be it Unix, Linux or whatever – I can use that platform to launch my virtual machines and get to work!

Getting rid of the laptop

Dragging your office to the office?

Now having to drag along a huge Powerbook laptop (i got the big screen dinosaur) and my external disks to work for a client is a drag. In most cases the client has a PC allocated for me, usually networked with a bunch of rules and also some restrictions on what you can install. I usually end up coding on my own laptop and then transferring the binaries to the host computer. Which is a giant waste of time and effort.

Since all I need is Linux and the VMWare player, it makes more sense that I create a bootable USB version of Linux and carry that with me instead. That way, I don’t need to carry my heavy laptop with me every time. I just need to plug the USB pen into the clients computer, restart and boot from that – and I’m ready to go!

I hope this post has made it more clear how I work and the benefits of using a virtual-machine for Delphi.

Using Linux with Delphi

In my previous article I explained in detail how to install Ubuntu Linux and get VMWare to work on it. I must confess that my earlier experiences with Linux were negative, but with Ubuntu that has changed. I moved from Windows many years ago to Mac’s and OS X, perhaps that is what has made Linux so attractive for me (OS X is actually Unix beneath, other older brother of Linux) but either way — Ubuntu is so user friendly and easy to operate that I cannot lie about that.

But I carry no i’ll will towards Windows. In fact, Windows is what I do. I know extremely little about Linux programming, but that is about to change 🙂 I have only good things to say about Windows. Stable. Secure. A solid workhorse (those tiles screwed up everything in Windows 8, let Windows be Windows and bring back the start-button and desktop).

I sinceryly hope I have helped someone to finding new technology and inspired us “old timers” to try something new.

Have a great day and god bless!

Moving to Linux – Part 2

Before you decide to try this, I want to urge you to be clever about it. First of all, make sure you back up everything. If possible, download Acronis and make a disk image of your present Windows Installation. Acronis is a fantastic tool which will pack your entire drive(s) neatly onto a series of DVD’s or external USB media, and you can restore your system back to its present state later without any hassle.

Secondly, always – and I mean always use the Live CD before you make up your mind. This is actually one of the great things about Linux, that you can boot from a CD and test the system first, without any damage done to your present operative system installation.

VMWare, clever programming

I must also urge you to move your development onto VMWare, not because you must – but because it will make your life easier in the future. If you are a Delphi programmer then this is the perfect situation to get this done, and it’s going to make your Delphi life so much better. Even if you decide to stick with Windows, creating separate Windows environments for each version of Delphi you own, getting to know the snapshot and clone tools — is going to revolutionize your programming. For instance, you set all your Virtual Machines to share a common source code repository (there is a time saver), which is so much easier when you work with different versions of Delphi and Windows. It also makes sense for your backup schedule, because you can back up everything in one go. Your Dropbox folder is a good place to keep your source-code and component packages (just a tip).

Delphi on Linux, you just saved $300 in License fees

Also make sure you clone each machine from a fresh install of Windows with just the essentials (virus killer, SVN client etc), and later reserve that clean root image for testing software. Before you install a program to test, take a snapshot of the virtual machine, test, then revert back to clean state afterwards. once you get the hang of developing with virtual machines, you will wonder how you ever managed to build programs professionally without them.

Creating these virtual machines will take some time, so allocate a day or two to get the job done. Get an external drive (2TB costs near to nothing these days) and back up everything there, both your vmware images and your Acronis disk image of windows. Remember to always have a secondary backup of your vmware disks. Should shit hit the fan, you don’t want to be left with nothing.

Selecting a Linux distro

On the face of it, Linux comes down to two factors: your hardware and your personal preference. My personal favorite is Fedora Linux; I don’t know why but I love the theme they use and how they organize things.  But sadly it doesn’t recognize my laptop (which I used as a “test” machine when trying out Linux). And this is why I hated Linux before – I was always missing a driver, or some part of my PC just did not work properly after installing. Which brings us to a third, more subtle factor, and that is funding.

The more money someone has thrown into a Linux distro, the more drivers and work will be invested in it. And it so happens that Ubuntu is maintained and evolved by a dot-com millionaire, which have thrown a ton of cash at the project (and it really shows). As such, Ubuntu is estimated to have roughly 3 times more drivers in their database than Fedora. If this is true or not I can’t say, but Ubuntu immediately recognized the built-in wi-fi on my laptop (even the on/off switch underneath it) so I can only presume the rumours are correct.

Ubuntu is an excellent Linux distro

So for this article I will focus on Ubuntu, because that is the distro which has done wonders for me. It is also a distro that is polished, spit and shine polished. Someone has clearly spent a lot of time and money to make sure Ubuntu is pleasant to use, that all the tiny details which makes all the difference (like two-finger scrolling) in daily use are in place, often associated with other operative systems (like OS X) or Windows. Basically, they have cherry picked ideas and details that makes sense from other systems – and combined them into an operative system which is a joy to use.

Downloading Ubuntu

If you havent done so already, head over to ubuntu.com and download the latest version (http://www.ubuntu.com/download/desktop). The download is an ISO image, so you need a CD/DVD burner and an empty cd for it. Once downloaded use Nero or any other cd/dvd utility to burn the ISO to disk.

Once you have the disk ready, reboot your PC with the CD in the drive (you may have to enter the bios and make sure the PC boots from the CD first). You will be presented with the option to either install Ubuntu or just try it out. I strongly suggest you spend a few hours playing with it before you install anything. This will also make sure Ubuntu recognizes the majority of your hardware. In my case, my gaming keyboard was not recognized immediately – and only when I installed it (and downloaded updates for the system) did Ubuntu find a driver for it, and all the keys lighted up, just like under Windows.

Installing

When you decide to install, make sure you check the “download updates from the internet” option. Ubuntu will compare the files on disk with the online versions, and use the online and more recent files instead. Since the install process is extremely quick and easy – there is not much to say about it. Make a cup of coffee and relax, you are almost there.

Starting for the first time

Once Ubuntu is properly installed, you are greeted by the Desktop. To a Windows user it may look strange. No start button in sight, a bunch of tool-icons to the left, and an Amiga like header at the top of the screen. Well, this is where the fun begins, because now we are going to install VMWare and get your Delphi environment(s) up and running again!

The Ubuntu desktop, more or less

In order to install VMWare we have to take a small trip to the shell. Click on the blue Ubuntu icon (on the button dock), this is actually the “Ubuntu Start Button”. But it’s a bit different. When you click it, a search window appears. At the bottom of this semi-transparent window there are some icons. Click on the second icon from the left. This is the “program” icon, which means that Ubuntu will display a list of all installed programs. If you type anything, this list will be filtered accordingly. Look through the list until you find “Terminal”, then drag that icon onto the button-dock so you keep it handy for later.

The Ubuntu start/search button has everything you need

Right, fire up the terminal by clicking on it and type in the following:

sudo apt-get install build-essential linux-headers-$(uname -r)

Tip: Just copy the text from this website, right-click on the terminal window and click “paste”.

Next, download VMWare Player and store that in “Downloads” (happens automatically), so click on the Firefox browser icon on the dock, and go to this website: https://my.vmware.com/web/vmware/free#desktop_end_user_computing/vmware_player/6_0. Pick the Linux download, either 32bit or 64bit, depending on your system. When the download is complete, you have to make the VMWare package executable. So simply open up the folder “Downloads” (you can do this from the browser), right-click the file, select “properties”, and check the “executable” checkbox.

And last but not least, let’s fire up the installer:

gksudo bash ~/Downloads/VMware-Player-6.0.2-1744117.x86_64.bundle

Note: You may get the message “gksudo” not installed, just follow the 2-line instruction on how to install it

Also, in the command above – make sure the filename (wmware-player-6.0. … etc) matches the file you downloaded. The version is no doubt higher by the time you read this. Also, Linux is case-sensitive (!)

VMWare installing on our spanking new Linux system

What, no automatic install?

If you are thinking that this is crap and that you were expecting an app-store like interface, don’t loose it. VMWare has chosen not to package their software, and Ubuntu cannot legally kidnap software and publish it. So VMWare has to be manually installed outside the ordinary app-store like system Ubuntu has to offer, but that is the only piece of software you need to do this with, unless you want some other non-packaged app on your system.

Either way — with VMWare installed you should now find the VMWare program icon if you click the “Ubuntu blue start-button” and select programs (second icon on the bottom of the window). Again you may want to drag the icon to the button-dock to keep it there, since you are going to use it daily 🙂

Going bananas on app-store

With VMWare in place you can now start to copy over your virtual machines (or perhaps you prefer keeping them on the external drive? Makes sense if you work in-office, then you just need to carry the disk with you in the morning). I presume you are not a complete computer novice and that you will have noticed the explorer icon light up when you opened the Downloads folder, so you wont have any problems finding the drive, copying the files to your documents folder and so on.

Now sit back, start-up your favorite Delphi and Windows virtual machine — ah! the joys of being a programmer 🙂

Now it’s time to go bananas on app store, which for Ubuntu is special for two occasions:

• 99.9% of all the apps are free
• It’s not called “app store” but rather “Ubuntu software center”

Download whatever you want, its free!

As a programmer you want something to edit graphics with in the line of Photoshop, so type in “gimp” in the search window – select it and click install. That’s really all there is to it. Everything is downloaded and adjusted automatically.

Secondly, you also want Lazarus, so type that in and do the same. Note: after installing Lazarus, search for “fpc” and install the dependency package (source files). Congratulations, you now have one hell of a compiler system free of charge.

If you are into 3d graphics I suggest you install Blender, and for audio there is audacity – which is also a great program on Windows. VLC is also a de-facto video program, a simply “must have” if you download .. ahem.. documentaries using bit-torrent (he said knowingly).

 Next time

In the next article on Linux we are going to focus more on common chores, like backup systems and how you can keep your Linux system and your data safe. We are also going to make a rescue USB pen-drive, with a full install of Ubuntu! In effect, the only thing you need to take to work – is that Linux pen + your VMWare machines on a secondary disk. Leave your bulky laptop at the office, you got everything you need to be a Delphi superhero on a stick!

 

Moving to Linux, why and how – Part 1

Linux. You either love it or hate it. And if you are like me, with a background on either Amiga or Mac, only to “grow up” and get a Windows PC for college or work – then chances are you have already tried several Linux variations over the years, but always ended up going back to Microsoft Windows.

My reason to use Windows is twofold: Working as a programmer for many years I have gotten to know Windows from the inside out. My primary programming language, Object Pascal, is strongly represented on Windows in the form of Embarcadero Delphi (previously Borland) – so simply dropping Windows over night was never an option. Dropping Windows would in effect mean putting myself out of work. So that’s not gonna happen.

Also, I don’t carry any ill will towards Windows. It has put food on my table all my adult life, so why should I hate that? I use Windows on a daily basis – but in my line of work it happens primarily through VMWare, not on an actual Windows machine. Hence I don’t want to throw huge amounts of cash out the window (pun intended) for an installation I quite frankly don’t use or need. As long as VMWare or any other virtualization tool is at hand, I can do my work.

Secondly, learning a new operative system takes time. It depends of course; on how much you need to know about something before you are comfortable using it. If you know every inch of Windows, and rate yourself an expert in all things Microsoft – then odds are you will resist going back to “newbie” or “lamer” status. People tend to stick with what they know, precisely to avoid feeling like a beginner.

If that seems about right, then keep on reading.

Elegant, easy to use and powerful

Mac’s are doing it

If you ask the average Mac user where the preferences files for a service is stored on his system, I doubt you will get an intelligent answer. The emphasis being “user”, not programmer in this context. The world of Apple is a purely consumer oriented enterprise. Users pay more money for a Mac exactly to avoid having to be an expert (!) In fact that’s one of the major selling points Apple got, that you can be a complete novice and still operate highly advanced technology.

A lot of very productive Apple users have absolutely no idea how Unix works. And Apple is doing their best to obscure the fact that they are selling Unix to the masses; Hiding the underlying filesystem, renaming important system files depending on the locale (localization of common system names) and much, much more. A Mac was never designed purely for technicians, hackers (in the original, kind sense), and under no circumstance for people who “think different”.

VGScene, before it became Firemonkey, on OSX

Be that as it may — the point of mentioning Mac’s was to demonstrate that you don’t have to be an expert in all things Linux in order to enjoy it. So what if you don’t know how to invoke some low-level kernel function (because calling kernel32.dll under Windows is what you do daily right?). If Linux has a program you enjoy using, which is helpful and makes you productive – why not make use of it? Why should we pay Microsoft $250-$400 for a system which, to be frank, you can’t trust. Windows Vista was a complete waste of money, only saved in the 11’th hour by Windows 7 (which made PC’s usable again). But as the smoke settled Microsoft went ahead and screwed up Windows 8 with “tiles”, trying to be something they are not. Leave iOS to Apple and stick to what Windows does best: the Windows desktop and Start button.+

Linux in 2014

If you had asked me some 10 years ago about Linux, I would probably have urged you to stick with Windows. My own experience was rather dim: always missing drivers. No games, only freeware crap. No Delphi, no Visual Studio – and you had to be an expert to uninstall a simple program. And should your disk get read/write errors — you were helpless unless you knew a Linux expert. That’s not a pretty picture. No wonder people have avoided Linux and stuck to Microsoft.

Play thousands of titles, nintendo, arcade, sega, psx on MAME

Today however, Linux is a very different system than it was back in 2004. Ubuntu specifically has financial backing, and the owner of the company have spent millions paying developers to write drivers and “fix” the package system. Installing a program under Linux has always been easy – but getting rid of it later (especially if something goes wrong) was a nightmare. For Ubuntu at least that is a tale from the past, and what you face now is — in lack of a better phrase, is a polished experience very close to Apple’s OS X. With one crucial difference: The majority of applications on “app store” are free – and installing them is done with a single click.

Virtualization

For the past 8 years most of my development has been done in VMWare, even on Windows. I own several Delphi version and have isolated each development environment in its own virtual-machine, complete with various versions of Windows. That way, I can create and test my programs on all versions of Windows from XP through Windows 8, using Delphi 7 through XE5. This setup, although time-consuming to make, have saved my bacon more than once. Especially when it comes to testing and bug-fixing (!) It also means that I can bring my machines with me on an external disk – no matter where I work. Many of my clients insist that I work “in office”, and spending days setting up their PC to match my development needs is no longer a problem. I just install VMWare player – and I’m ready to code.

VMWare on Ubuntu

The benefit of all this is that the underlying operative system doesn’t matter as much as it once did. As long as I can install VMWare – I can use Windows, OS X, Linux and even Spark Solaris for that matter (!). For me personally it means I save quite a bit of cash, because I don’t need to pay for more Windows Licensenses. I own the licenses for my Virtual Machines — and that’s it.

As a “Delphi” programmer I can also enjoy FPC/Lazarus natively on Linux, and generate binaries for Windows, OS X and Linux from the same IDE. Perhaps one day Embarcadero will offer Delphi for Linux and Mac, but until then I don’t have to pay thousands just to do my job (!).

When it comes to everyday tasks Linux also suits my needs. There are a ton of Open Office forks out there (libreOffice is very polished), email clients, browsers, music players, dvd/movie players to choose from. And should I feel the urge to play games then I can just fire up Steam – or Windows in VMWare and play whatever I love.  I’m more of a retro gamer so I enjoy MAME, Scumm VM and Amiga Emulator’s more than I do modern games (I own an XBox and PSX1-3 so I don’t suffer in that department).

 Getting started on Linux

If you fancy getting started with Linux as a Delphi programmer then this article series will be for you. I will go through everything: getting your Delphi installation into VMWare, installing Ubuntu on your machine – and installing VMWare on your Ubuntu installation. We are also going to dig into Lazarus and various other development tools, not to mention getting Smart Mobile Studio to run under Linux so you can continue to write cutting edge HTML5/JS applications.

Suck on that Bill Gates 🙂

In the meantime, download the latest Ubuntu distro and burn it to a DVD/CD Rom. Spend a few minutes making sure you have backed up everything on an external drive (Fat32 or NTFS formated) so you are absolutely sure nothing can be damaged if you do something wrong. Dropbox is a neat place to store your source files 🙂

Converting QTX Pascal to FreePascal/Lazarus

As you may remember, I run a second website for Quartex Pascal, a research IDE for custom compiler technology. Aiming first and foremost to be the object-pascal equivalent of Eclipse. In many ways QTX continues where Smart Mobile Studio ends. Smart Mobile Studio targets HTML5 exclusively – aiming to be the best Object Pascal to JavaScript compiler on the marked — but Quartex Pascal is a very different beast all together.

The QTX IDE has so far been written in vanilla Delphi XE2 and XE3. It uses plain generics and is otherwise without external dependencies. The editor functionality is provided by a forked version of SynEdit, and as of writing the DWScript engine is the primary code generator framework. But depending on how portable DWScript is (a port of DWScript is reported to be in circulation, but if it supports the code-generator framework added by Eric Grange is unknown at this point) that will probably change.

There is also a secondary code generator framework being written for it, which uses Castalia for parsing and building the AST (abstract symbol tree). It is also capable of parsing visual basic .net (OOP, rule based parsers are very easy to write using my framework). The parser/tokenizer constructs an AST using an intermediate language format, which in turn can be stored as plain XML. This is quite revolutionary because it means – that turning the intermediate code into “real” code is now a matter of XML transformation, which is an off-the-shelf technology tried and tested by time. Ultimately, QTX may use the freepascal universal compiler itself to build “real” applications, or custom code generators – targeting mono C#, GCC or any other native compiler available on a target system.

Porting to freepascal

In order to support more platforms I have decided to port my work wholeheartedly to freepascal. This does not mean that Windows wont be supported anymore, or that I’m abandoning ship – far from it; it means that QTX will eventually be available for Windows, OS X, Linux, Amiga OS and MorphOS to name a few.

I will no doubt continue to work with Delphi for a long time, but I must admit that I am beginning to phase out Embarcadero from my toolkit, in favour of FreePascal, JavaScript and C#. Delphi’s pricing has become ridicules, the generated code is not what it used to be – and since FreePascal and Lazarus have reached a point of quality that is hard to rival — I see no reason continuing to pay top dollar to a company so clearly hostile to Smart Mobile Studio. In fact, they threatened to poll their funding for a “famous Delphi event” if Smart Mobile Studio was allowed to be presented — which is just plain unacceptable behavior for a company like Embarcadero.

If you have never tried Freepascal + Lazarus under Ubuntu, I sincerely urge you to install it. It truly is a fantastic experience!

My desktop right now

 

Pascal community project: Thundercloud

The G4 PPC Mac, you must admit that it’s a sexy piece of machinery

If you follow my weekly ramblings you know that I’ve spent some time recently with hardware; Recycling older hardware for new and modern purposes to be precise, with focus on PPC Apple Mac’s and older Intel PC’s.

I am still in shock over how people treat these fantastic machines. In one case, a man wanted 300NKR ($25) for a G4 Mac – which is basically the same as saying “if you pick it up, it’s yours”. That very same machine cost roughly 16.000 NKR some 14 years ago. It was my first introduction to Apple products and I used it for music production. So I know a thing or two about using these machines for high performance tasks. I realize of-course that 10-14 years is a long time in the world of computers, but once again it depends on what you actually want to achieve.

I mean, the Raspberry PI is a fun kit to play with, but it’s a three-wheeler for toddler’s compared to a fully specked out G4 or G5 Power Mac.

What’s the point with all that hardware

There is a limit to how many computers you want in your house. My goal of 5 G4 Mac’s and 5 G5 Mac’s inter-linked by optical connections is already raising eyebrows with my girlfriend – so I may have to move my DIY project into the basement (I sort of predicted that *smile*), but the facts of the project is simple:

• Collect enough CPU power to host a cloud
• Attempt to interface or write the cloud architecture in FreePascal
• Use either native OS X or Ubuntu for PPC

Each computer will be designated a specific layer in the personal cloud. For instance, a friend of mine donated his previous gaming PC to the project. But it turned out to be anything but old (well, it’s a few years old but the hardware is en-par with great systems even today). It contains two quad-core 2.4 GHz Intel processors (read: eight cores), two NVida G-Force graphics cards which are inter-connected – and much, much more. In other words, this machine is capable of running Crysis-II on two monitors in full HD without breaking a sweat!

The logical place in a cloud for that piece of hardware is to host the video conversion API. Due to the number of cores and powerful GPU’s this is a natural place in a cloud architecture for this particular system. While the Mac’s are fun and cool – their graphics cards cannot compete against that kind of power on their own. The graphics processing power of that one PC probably equals two or three G5 machines (plain hardware, no extended graphics card).

Ubuntu

I feel I have to write a bit regaring Ubuntu. My favorite Linux distro has always been Fedora Linux (a Redhat community project). I don’t know why, but the Fedora visual styles and the way the system is organized appeals to me. Sadly, Fedora has a smaller driver-database than Ubuntu – so getting Fedora to properly recognize my old laptop has been disappointing to say the least. The built-in wifi is not picked up and things like camera and card-reader is rendered useless by Fedora.

The ubuntu desktop

I decided to try Ubuntu and, as expected, it immediately recognized the hardware and every feature, which was a much more pleasant user-experience. It took roughly one hour to install and update the system, but once Ubuntu was properly setup (which was just a matter of selecting language, creating a user account and setting a few options) I found myself in a polished. modern and capable desktop environment.

As I began to investigate the concept of personal cloud computing I realized that Ubuntu pretty much contains what I was looking for to begin with: namely a fully working personal cloud solution which is free, international and open-source. Ubuntu uses a system called OpenStack, which is a cloud system with support for a variety of programming languages. The typical candidates are of course Java, .Net, Python, nodeJS and more (you know, the typical stuff). But there was no support for Object Pascal – which came as no surprise.

But, if C/C++ can do it — Object Pascal can do it, so I’m hoping to convert the C/C++ headers to FreePascal. As such the native, portable, platform-independent “ThunderCloud” project can become a reality.

What exactly do I mean with “Cloud”?

I must stress that the word “cloud” is being hyped by spin-doctors to mean either very little or very much. But it does not simply mean storage! A lot of people look at the cloud phenomenon as a kind of anonymous giant filing cabinet in the sky, where large and small files are loosely connected to an account, only to magically disappear when you delete something.

Cloud

The concept of cloud ranges from being able to instantly clone and run full virtual-machine images (think vmware environment), scale them to any number of instances, create virtual networks between these and real computers, manage terrabytes size databases, backup solutions, email, and last but not least – to manage all the software on the images, no matter how complex they might be.

Cloud is also an open, account based, API for the web. Providing a great number of API’s for web developers and native developers over a wide variety of protocols and standards (REST, JSon, XML, noSQL, SQL-Server, mySQL and high-speed file storage). It is this latter part of a personal cloud that I am interested in:

• User account management
• Backup services (Mac, Linux and Windows)
• Media streaming and conversion
• File management
• Message handling (email, messenger, sms, IP telephony)

Lazarus on Ubuntu

Say what you will about FPC/Lazarus but it really comes into its own on Linux. Under OS-X Lazarus feels.. well, “out-of-place” is one way to describe it. I havent tested Lazarus under OS-X lately, but when I did use it a couple of years back it was haunted by terrible bugs, strange crashes and inability to adopt the native OS-X theme. Problems made even worse by the fact that Mac’s treat you as a child, hiding and translating filenames en-mass to obscure the fact that you are actually running a Unix system. I pray this has changed by now, and that Lazarus/FPC is just as elegant to use as the Linux version is.

Lazarus on Ubuntu

I think this was the very first time I have ever installed and started Lazarus without something being wrong. The IDE asked me to locate the source path at one point, but after installing a couple of dependencies from the Ubuntu software app-store Lazarus was content.

With the exception of the “Delphi 1 .. 7” look, which I absolutely hate, Lazarus was a joy to use. And this is the first time I have used such a description on Lazarus, because all my previous experiences with Lazarus (OS-X for the most part) have been tragic and negative.

With the FreePascal compiler supporting multiple targets; you can compile for Linux under Windows, and for Windows under Linux — I am now questioning my use of Windows for personal computing. We do use Windows at work – in fact, it’s a 100% Microsoft business from beginning to end, so I wont be able to dodge Windows licensing there. But what I do with my spare time and what system I use at home, that’s my business and nobody else’s.

I must admit that the only reason I havent switched to Linux before is because I relied on Windows for Delphi programming. But with the quality of Linux now starting to rival Mac’s, and at the same time the quality of Lazarus and FreePascal rivaling that of Delphi — I see little reason in paying Microsoft or Embarcadero for their services anymore.

C# under Ubuntu works like a charm

Linux also runs MonoDevelop, which I use for C# development so once again I question the place for Windows on my personal computer(s). And should I for some reason need to run anything exclusively Microsoft – Wine and VMWare pretty much have it covered. But I cannot for the life of me recall what program should be so important that Linux can’t provide identical or better alternatives. Photoshop has become ridicules over the past 8 years, so I actually prefer “the gimp” for giving the same amount of features at only a fraction of the size and complexity.

Blender runs perfectly under Linux as it does on Windows, same can be said for webkit browsers.

The only program which I presently cannot run or work on, is Smart Mobile Studio (which is a must), so I installed VMWare in order to run my development environments separately (just like I do under OS-X and Windows). I always use VMWare for my Delphi work. Each version of Delphi is installed into its own separate environment, using several different operative systems (Windows XP, Windows Vista, Windows 7 and Windows 8). This makes it easier to work, debug and test. On all platforms.

 

Object Pascal, power computing at extreme budgets

Being a programmer in 2014 –having been a coder through the 90’s and 2k’s until present day – is like living in disney land. When I was a teenager my most priced possession was my Amiga computer. It cost a fortune and was powered by a whopping 1Mhz MC68000 CPU (yet due to it’s custom chipset, it outperformed PC’s up until GPU powered 3d cards became standard around 1995). The Amiga was the bomb back then. And it’s memory capacity was jaw dropping 2 megabyte on-board chip-ram. I extended it with 8 megabyte additional ram, which set me back around $400, which for a poor student was a small fortune.

What can you get for $400 today? Well today you can pick up a second-hand Apple Mac G5 Dual-Core PPC for less than $200, complete with a decent monitor. And if you know how to use bit-torrent, you can pimp that machine so full of software that it has more value than a new $1995 iMac. Photoshop, pro-tools and gcc / x-code was awesome in the PPC as well, not just the Intel Mac’s.

Object Pascal on older hardware

Lately I find myself thinking that it has to be possible to re-cycle some of these old machines and apply them to new and modern purposes. I mean, a G5 PPC mac is a processing beast compared to it’s contemporary PCs 8 or 10 years ago. The G5 dual core processor was the final evolution of the PPC range of processors – so it’s a fine and powerful piece of engineering for a ridicules price. Making use of these machines with object pascal sounds both fun and interesting.

With this in mind I did a quick search on Finn.no, which has a second-hand market; comparable to craigslist, ebay and sites of that genre. And the average price for perfectly usable, good condition second hand Apple G4 and G5 Macs were in the 500-100 NKR price range. Which is roughly between $100-$200! That is a lot of CPU power ladies and gentlemen, in fact, it’s almost sad to see these machines which look more like works of fine art than computers being practically thrown out the door for the price of a playstation game.

MorphOS looks like a blend of Amiga / Linux meets Windows

So what? I hear you say. No modern software will run on these machines – so they are useless. It’s just a heap of unusable iron taking up space.

Well, not quite. Linux happily runs on PPC hardware – some users argue that Linux runs better on these machines than Mac OS classic and OSX did to begin with. But if you are adventurous and able to mentally disregard the OS factor (for now) and would like to use Object Pascal for specific work related tasks, then you can include an alternative operative system called MorphOS.

Pimp up old hardware with a small, compact and lightning fast OS

Now before we continue, let’s look at a couple of tasks where old hardware can be recycled and which supports object pascal (freepascal and/or lazarus in a desktop environment). And lest we forget, there is a fork of lazarus called Code Typhoon which is rarely talked about yet enjoy a steady following around the world due to it’s stability and rich component base. And it’s free (!). But first, let’s look at some tasks suitable for re-cycled hardware:

Web server

The most common task for older hardware is, naturally, to be used as a vanilla web-server. Apache (the webserver for linux and other alternative operating systems) is for the most part hardware agnostic –and as long as the latest berkley tcp/ip stack is installed, compiling Apache from source is easy and hassle free (3 lines from the command prompt under linux/unix).

If you add NodeJS to the mix then you have yourself a paradise for Smart Mobile Studio development, since Smart Pascal allows you to write both client and server from the same codebase, running the server code under nodeJS and client in any HTML5 compliant browser. So building your own client/server environment for testing purposes at home for less than $200 is more than possible.

Note: nodeJS may not run on older versions of OS X, but it will almost certainly run if you install linux.

Backup server

Another form of server which is a must these days both at the office and in your home, is a dedicated backup server. A lot of people fork out $400-$600 for a network disk or NAS server (both for backup and movie streaming), but that’s actually a complete waste of money; Because you can as I mentioned, pick up a second-hand PPC mac for 1/6 of the price which is 50 times faster and with plenty of room for IDE disks (not to mention remote desktop options so you can control it from your work PC or Mac). OS X also have functions setting up a machine as a backup device for other computers – and a fileshare on the network for movies and music is a matter of flicking a switch.

Education

If you work as a teacher, pupils dont really need access to the very best. In fact, learning to program in Lazarus on a second hand PC or Mac (the latter recommended) running linux or OSX is a fantastic way to broaden the pupils horizons. Delphi has become extremely large and heavy in terms of technology. Beginners without proper documentation can quite frankly get lost in Delphi XE 1 through 7. So starting with Lazarus and freepascal, which is a delight to use on Fedora Linux, is an excellent start!

A good object pascal programmer could make a network program for tests and exams, which makes sense for schools on a budget. You dont have to fork out thousands of dollars or pounds for a uniform computing environment + software.

Store front-end

If you work in technical retail your store-front is bound to run presentations, video and/or demonstration slideshows. It can cost as much as $2000 to buy a professional multimedia studio, adapter packages and cables, not to mention database integration for daily updates of prices and offers. With the help of Freepascal and a dedicated machine, an old iMac G5 is more than powerful enough to handle a full shop front-end, multiple monitors (chained even if you like) and/or database presentations. Graphics32 (which has been ported to Lazarus/FPC) makes effect programming extremely easy – and you can throw in openGL if that tickles your fancy.

Paint mixing and customer kiosk systems

Paint is one of those items which everyone uses, but we rarely think about it until we need it. Most stores that ship and sell paint have digital mixing programs where the customer can select and adjust (create) their own blend of paint. It’s actually a very easy application, at least those that i’ve seen, which I would complete in roughly 2 days work. The only time-consuming task for such a project, is coding the serial-cable protocol for transferring the RGB color values to the mixer. Again — old hardware is up for the task. A PPC G4/G5 is ample power for running a fullscreen, mouse driven, object pascal application — and Freepascal is very well evolved so you will find everything you need in the RTL.

Media Server / Center

A $30 Raspberry PI mini computer is more than enough to power the latest Linux media station (or server) software. Since Linux is extremely popular you will also find the latest versions of Freepascal and Lazarus in most distro’s (including Raspberry PI’s repo). If you don’t want to fork out for an Apple-TV or Google TV stick, then you can easily build one yourself with a raspberry pi.

Build server (SVN)

While perhaps a bit overkill for the lone hobby programmer, it can be a valuable exercise for professional and amateurs alike. I personally have my personal SVN server running and use that in combination with a backup server to keep my 15+ years of code safe and up to date. Using an old G5 or G4 to maintain your company source-code (if you represent a team of 2-10 programmers) is not just good practice – it’s a required minimum.

And if you are thinking, how can I do nightly builds on a PPC machine? Well, if you use freepascal then that wont be a problem (multi target compiler). But if you use Delphi you may want to run Windows under Bochs (a bare-bone windows XP is enough if all you want to do is compile):

• Bochs for OS X (PPC): http://www.macupdate.com/app/mac/11177/bochs

Selecting hardware

If you can get your hands on a second-hand Apple iMac G5 workstation (PPC processor), which in Norway at least can be picked up for around 1000 NKR (US $200) that is an absolutely fantastic machine. It is also “modern” enough to run a good selection of alternative operative systems (alternative to Windows and OS X) as well as OS X up to version 4.5 if memory serves me right. This is a perfect machine to recycle for new tasks.

Pick up a Power Mac G5 for next to nothing

Apple G4 machines, which I find esthetically pleasing and fancy (and easy to fix, replace parts and code on) is also a good find. But you should make sure you check the hardware against the MorphOS hardware compatibility list — which is also a good list for Linux (to check if your old machine can be used with modern Linux distros).

Older PC’s is likewise perfect for recycling, but once again you should check for driver support (which is always the problem with PC/Win machines, as opposed to Mac’s which have a fixed chipset). I would not buy a PC older than 7 years, and would probably pay very little for such a machine ($40/50). If you buy or re-cycle an older PC, make sure you have at least 4 gig of ram (the max for WinXP unmodified) if you plan to run Windows, same goes for Linux.

Lazarus + FPC is best enjoyed under Linux, here running in Fedora

At the very lowest end of the spectrum, but surprisingly the most fun to play with – are embedded micro computers such as the Raspberry PI. Starting at $30 it comes with a quite powerful GPU, making it ideal for homebrew media center projects. It supports Linux and as such it has full access to Object Pascal/ freepascal. But, due to the very small processor, Lazarus is a bit to slow for serious work — but FPC/Lazarus executables run very fast and is in my view the best language to use; side by side with C++. If you combine FPC with SDL (simple direct media layer) you have a pretty modern multimedia engine to play with, regardless of CPU and architecture.

Lazarus + FPC running on Raspberry PI micro computer

What about the web

The downside of working with older hardware is that you can only use them for development. The moment you want to ship a product written in platform independent object pascal you have to get your hands on a machine from the modern marketplace. But there is one combination where you can avoid all that – and that if if your target media is HTML5 exclusively.

While Delphi XE 1 through 7 is far to processor and memory hungry for an older PC, Smart Mobile Studio is absolutely perfect. It comes with a small and compact RTL for making cutting edge HTML5 mobile apps. It has a chrome browser built-in (embedded) and is more or less everything you need to write JavaScript based applications designed for either web-pages (embedded like a flash app would) or a fullscreen mobile app. You write object pascal, the compiler generates hardcore JavaScript from that.

And JavaScript is extremely fast, in many cases (especially when it comes to graphics) faster than native Delphi (which sounds ridicules I know, but check the benchmarks and see for yourself). JavaScript also has the benefit of running pretty much anywhere in a modern browser.

Get away from platform bound code with Smart Mobile Studio

So one very cheap alternative is a 5+ year old PC with Windows XP setup just for Smart Mobile Studio development. I actually have several such machines setup, both real hardware and virtual machines (VMWare).

• V8 JavaScript engine for PPC (OSX): https://github.com/andrewlow/v8ppc
• NodeJS for PPC (OSX): https://github.com/andrewlow/node
• Bochs X86 Emulator for PPC MAC: http://www.macupdate.com/app/mac/11177/bochs

Final verdict

Is it possible to build your own fantastic object-pascal super computers on an extreme budget? 10 years ago the answer would have been a loud “No!”, but today the reality is that you can buy extreme amounts of processing power second-hand for next to nothing. And you can make use of FreePascal and Lazarus to build custom systems – systems which can be re-compiled on more modern hardware when needed. This makes for some very interesting cross-platform solutions.

I should also mention that a lot of virtual machines, like Bochs (free) run perfectly fine under PPC hardware, meaning that you can in fact setup a test-environment for your Delphi and/or freepascal projects on an older Mac – and just remote desktop your way into the test environment whenever you want.

And one scenario I forgot: Your own SVN server is also a good use of old hardware.

Well, I hope you have found some inspiration to re-cycle technology and put your object pascal knowledge to new and exciting uses. Who knows, perhaps you come up with a good idea and can ship out 100 used macs preloaded with your software?

Nothing is impossible 🙂

QTX RTL, experimental Smart Pascal run-time library

The Smart MobileStudio RTL I designed primarily for mobile applications, and it does a damn fine job at managing that – but what about larger and different form-factors? What about applications designed for a full desktop browser? Or even better, compatibility with native Delphi?

Well this is essentially what the QTX RTL is all about: To have a platform for RTL experimentation and evolution, a kitchen-sink where we try stuff out before it’s released into the wild and merged with Smart Mobile Studio’s RTL.

Advances

Since I have just begun architecting this one there is still a lot of work ahead, but here is the general class hierarchy:

• TQTXErrorObj
  • TQTXPersistent
    • TQTXHtmlElement
      • TQTXComponent
        • TQTXControl

That might not seem so different from the VJL (Visual JavaScript Component Library) as deployed by Smart Mobile Studio today, but there are many important design considerations involved. Most notably:

• Built-in error management at the core
• Built-in persistent mechanism (RTTI is optional)
• No dependencies on low-level procedures
• Delegate support
• Interface driven

Delegates

This is a topic that must be explained a bit more in-depth, as it is extremely important and beneficial. Delphi and other Object Pascal compilers dont use delegate objects, they use what can only be called “ad-hoc” event handlers. In essence, Delphi events are nothing more than a bare-bone callback mechanism. This is extremely fast and effective, but limited to only one receiver at any given moment.

A delegate however, is an object wrapper around a classical event handler. You could say that it represents an object-oriented version of the ad-hoc mechanism. And it comes with the benefit you would expect from OOP, namely that you can create as many object instances as you wish. In other words, you can have 10 delegates responding to the same event if you so choose (!).

This is extremely efficient for component writers, because descendants can now respond to the same events as their ancestor methods without interfering or having to override anything. You also get away from the classical Delphi-style exposure of event trigger procedures (like below), which is quite frankly unnecessary bloat.

Procedure TMyComponent.DoChanged;
Begin
  if not (csDestroying in ComponentState)
  and assigned(OnChanged) then
  onChanged(self);
end;

Another nice feature of Delegate objects, is that they are resource efficient. If you only listen for one event, then no resources or callback handlers are allocated for anything else. Delegates also have the benefit of being persistent, name based (you define the target event by name, not slot or id like under OSX or GTK#). And perhaps more importantly – they are natively supported by all browsers and as thus represents a very thin layer over the actual HTML5 framework.

Future IDE advances

There are also quite a few benefits if we consider the Smart Pascal IDE. Should delegates be introduced in the form-designer, it would to a great extent remove the need for event inspection (which is a fairly complex process). You would simply add the number of events you want for an element, perhaps more than one foe the same event if that is necessary – something which would is infinitely easier to deal with.

Delegate examples

Creating and using a delegate is almost as simple as setting an event in Delphi or Smart Mobile Studio today:

var
  mDelegate: TQTXDelegate;
begin
  mDelegate:=FControl.Delegates.add;
  mDelegate.Bind('onClick');
  mDelegate.onExecute:=Procedure (EventObj:TQTXDelegate)
    Begin
      //Handle event here
    end;
end;

Or the short-hand version:

var
  mDelegate: TQTXDelegate;
begin
  mDelegate:=FControl.Delegates.add.Bind('onClick',
    procedure (EventObj:TQTXDelegate)
    Begin
      //Handle event here
    end);
end;

Triggering a delegate can now be done several ways. First, if you have access to the delegate object itself:

  mDelegate.Dispatch;

And secondly by name, which will only dispatch the event if it’s actually has a registered delegate(s) for that event:

begin
  //Trigger all delegates mapped to onClick
  FControl.delegates.dispatch('onClick',[0,1,"hello"]);
end;

And finally you can mute events with a one-liner:

begin
  FControl.delegates.active:=False;
end;

And last but not least, storing delegates is a piece of cake:

var
  mJSON: String;
  mAccess: IQTXPersistent;
begin
 mAccess:=(FControl.delegates as IQTXPersistent);
 if assigned(mAccess) then
 mJSON:=mAccess.writeContent;
end;

So finally we can decouple event management from the work of writing components. Such things can now be regarded almost as action objects in Delphi, specifically tailored and isolated in classes.

Assasin creed hoodies, scam?

Updated 30.12.2015: The website has now simply changed their domain name to leogary (http://www.leogary.com/) but be aware: Its the same scam and the same company!

A few weeks back I came across this website: www.assassinshoodies.com, which sells some alternative and cool looking hoodies, sweaters and jackets based on the in-game design from Assassin’s Creed. So I ordered two pieces of size L hoodies for my 12-year-old son. Since sizes differs across borders I would have to order an XL for myself if I wanted one — but my son loves the game and he was so looking forward to getting “real” assassin creed hoodies so I went ahead and ordered.

The odds of missing the size, sending a completely different low-quality brand – and missing the postal code is in my view extremely low. It’s like ordering Italian Pizza and getting a kebab delivered next door from a place you never called.

As you can see from the picture below, the hoodies doesn’t actually have a hood. They have a high neck with supported brim, which is perfect for Norwegian weather in the fall / autumn, add a small scarf inside the neck and you can safely go jogging or send your kid to soccer practice in cold weather.

No hood for the hoodies

What did we get?

First of all they managed to send us 2 pieces of hoodies size XXL, meaning that they dont even fit a grown man, let alone my 12 year old son! Secondly they sent it to the wrong postal-code, so I had to drive 40 minutes to pick them up.

But last and perhaps more importantly — they sent us two ordinary hoodies of a completely different brand (!)

A plain, sad looking, grey hoodie – with a hood! XXL

The hoodies are plain old sweater-hoodies, low-quality fabric, marked “MIAO DU”, which after googling it seems to be a production company creating just about everything under the sun in China.

The odds of missing the size, sending a completely different low-quality brand – and missing the postal code is in my view extremely low. It’s like ordering Italian Pizza and getting a kebab delivered next door from a place you never called.

So while the company I ordered from is American, their production and storage is kept in China. How many people do you think bother to return a $80 order which took 5-6 weeks from China — back to China after being ripped off? Might as well throw the money out the window, twice!

So I urge everyone to not order a single piece from http://www.assassinshoodies.com. I was under the impression I was buying from an American company which could be trusted, but in fact everything seems to take place in China. And I don’t order things from China due to past experiences with false information, scams and an utter lack of respect for customers.

Be warned — stay away!

Castalia parser, how to use

Everyone who has ever wanted to create their own scripting engine, or indeed – their own compiler, have checked out the free and open-source Delphi parser called Castalia. Written by Jacob Thurman, the Castalia parser is also a vital part of the Castalia Delphi IDE extension. If you havent had a look at Castalia before then head over to JT’s website and have a look.

To sum up the situation: Castalia is a commercial product, but the Castalia Parser is open-source and can be used, improved and downloaded by anyone. In fact, the Castalia parser is hosted on GitHub here: https://github.com/jacobthurman/Castalia-Delphi-Parser

So why isn’t it used in more projects? Well, because it’s hardly documented. And examples are thin on the ground to say the least. But perhaps more than anything else — parsing programming source-code is not within the range of “normal” programming. There are a few good books on compiler and parsing technology out there, but in general this is something you either learn through advanced computing classes at some university (or just figuring it out by spending time on the subject). So it takes a bit of thinking before you jump in.

So how do we use it?

Simple, but before we dig into the how you have to understand why. Without understanding why a parser is designed the way it is – you wont really be able to use it properly. So here are some simple factoids to teach you just that:

• A parser reads a file character by character, not line by line like humans do. To a parser, CR(carriage return, #13) and LF (linefeed, #10) has no meaning at all, they are just characters like any other. It’s up to the parser to treat these are line-end identifiers.
• A human readable word, like “begin” or “procedure” is simply “an array of chars” with no meaning. The meaning is once again provided by our code.
• Connected to the parser is something called a “tokenizer”, which is a class meant to translate and recognize words and characters, turning them into tokens – which we can work with in our programming.
  So characters like “(” becomes “OpenRound”, and “;” becomes “semicolon”. A tokenizer is the computer version of what human users recognize as the language being parsed.

A tokenizer is what makes it possible for us to write criteria and expectation code. Expectations are the criteria the parser holds for a piece of code, these are called “future” elements, while tokens who are already read are called past criteria.

Take something simple, like parsing the following:

Procedure TMyClass.Myproc(aValue:Integer);
begin
end;

The code for dealing with the keyword “procedure” will have expectations to what comes next. Namely that it’s a valid procedure name. It can also have criteria to the past, for instance – is it defined inside a class? If that is the case, then the class-name should be omitted and no begin/end section is allowed (that should be below the implementation marker).

Moral? Parsing is just as nitty-gritty as you imagine it to be!

With that in mind, have look at the Castalia parser source-code. You will notice that each language word or feature is mapped directly to a class-member. What you are supposed to do here is to override those that you need (a full engine would override them all) and use the information to build a program-model in memory (a program would be a list of units, who contains a list of classes and functions, constants and variables etc).

What you must take height for, is that elements are not created complete, but one piece at a time. A class declaration for instance, should be registered in your model when the class method triggers (you have to check that it’s not just a forward declaration ofcourse), while the name of the class is delivered afterwards as the parser get’s to it. So you need to write you parser like this:

type

TMyParser = Class(TSimpleParParse)
private
  FCurrentClass: TMyClassDef;
  FCurrentMethod: TMyClassMethod;
end;

So whenever the class is invoked and it should be registered, you register the object without a name and assign it to FCurrentClass. Whenever a class method is triggered, then it should apply to the current class and also be assigned to FCurrentmethod so setting the name is easier. And so fourth. This applies more or less to all the constructs of the language which Castalia covers and deals with nicely!

You could, perhaps, build a simplification object on top of castalia, which triggers events like “OnRegisterClass(Sender:TObject; Ancestor:String; aName:String)” or something like it, but Castalia really is extremely well made and simple to work with once you understand how it works.

Language rules

But now you probably wonder — what about rules? If the parser just blindly recognize words and triggers that procedure in the Castalia parser class — what will protect us from translating gibberish? I mean, would “begin procedure function” actually compile?

No. The rules of “object pascal” is maintained by two things:

• Expectations to future words (read: symbols)
• Expectations on past words

For instance, a method dealing with “procedure-names” would expect the past word to be “procedure”, and it would also have expectations on future symbols to be either OpenRound (followed by a list of parameters) or just semicolon.

  procedure SOMENAME( params );
  procedure SOMENAME;

Compile sequence representation

If the expectations on future symbols (being either “(” or “;”) is not met, then you know something is wrong and it’s a syntax error. And it’s the same with expectations on past symbols. If the past word is not “procedure” or “function”, then once again we know that something is wrong and throw a syntax-error exception. If the past symbol is “function”, we would also expect “:” and “datatype” at the end of the function declaration. So parsing a procedure is a bit different from procedures. It also defines if the built-in variable “result” is allowed in the content code.

Either way — When you combine the complete set of expectations for each keyword — the end result is what we call object pascal, and it defines what you are allowed to do in every aspect of the language. You dont need a huge kit of code to test for everything under the sun first, you test the validity of the program one symbol and token at a time.

To read “future” tokens you use the following:

Lexer.InitAhead;
FNextToken:=Lexer.AheadTokenID; //What is the future token?

The extended ID of the “present” position can be read using the TokenID and ExID properties. There really is no need to document these features here, since they are used everywhere throughout the source code. You will also find how to use them in the actual methods you want to override. And last but not least, expectations are handled by “Expected(ptSemiColon);” on entry (replace token by what you expect).

What typically confuses people when playing with Castalia, is the utter lack of an object model. Which is not the job of a parser/lexer/tokenizer at all.

Object model, the missing piece

What Castalia is, is a very effective hand-written parser for Delphi. The author(s) really should be praised for this – and it’s very sad that it havent been deployed in more projects over the past 5-6 years. It’s like a gem that has been lying there without being noticed like forever, waiting for someone to pick it up and make use of it. But it’s missing the piece which naturally is what people want, and that is an object-model for a unit of program.

Such an object model is typically called AST, which is short for “abstract symbol tree”. How you implement and organize this piece of code is different from language to language and compiler to compiler, but a few common denominators are to be found across the board.

The AST is basically what the .NET “intermediate language” compiles from. So the first pass of the .net compiler breaks the source-code into an AST, the second pass takes the AST and compiles it into CIL code (virtual assembly language) – which at it’s final stage can be translated into real machine code.

If you think this is easy work – think again, because finding a model that is flexible enough to work AND be maintained over time, is extremely hard. In fact, it doesn’t get much harder in life than writing compilers (with the exception of coding a kernel or OS perhaps).

Take something simple, like optimizing the generated code, let’s say we have a procedure which looks like this:

procedure TMyObject.SomeProc;
var
  x: Integer;
Begin
  for x:=0 to 255 do
  inc(y);
end;

A silly example I know, but could this be optimized? Yes indeed. By doing loop expansion we could handle 8 calls at once or 16, that is a typical optimization to better performance. It has the benefit of being simple to automate and make work without side effects (if the conditions are right):

procedure TMyObject.SomeProc;
var
  x,y: Integer;
Begin
  for x:=0 to 31 do
  Begin
    inc(y);
    inc(y);
    inc(y);
    inc(y);
    inc(y);
    inc(y);
    inc(y);
    inc(y);
  end;
end;

But in order to do this automatically, the AST have to well enough built. Remember, what we are dealing with is an abstract symbol tree. So in reality it looks like this:

• [TQTXClassMethod]
  • [TQTXClassMethodLocalVariableList]
    • Items
      • X
        • Datatype: TQTXDatatype.dsInteger32
        • Name: “X”
        • InitialValue: [QTXConstantValue] 0
  • FOR
    • Target: [QTXClassMemberLocalVar]
    • Start: [QTXConstantValue] 0
    • End: [QTXConstantValue] 255
  • INC
    • Target: [QTXClassMemberLocalVar]
    • Value: [QTXConstantValue] 1

In this case, the inner expression-block consists of just one “INC” call, so we could perform loop expansion by simply duplicating the expression-block X number of times (8 or 16 being traditional numbers) and get away with it. But once the internal expression-block (the code within the for/next loop) is to complex, that wont work. In fact, it may screw up the entire program.

A second optimization would be to remove the whole for/next block and replace it with a single:

• INC
  • Target: [QTXClassMemberLocalVar]
  • Value: [QTXConstantValue] 255

That would be the next best optimization. But since we are using inc() on a local variable only to exit the procedure, the procedure should be ultimately eliminated from the compiled code altogether, because it doesn’t alter existing data or produce any output. This is an area of compilation where DWScript is really, really good. It’s a perfect example of a seasoned, evolved and well designed compiler and code-generator. Eric Grange really did a fantastic job on that one.

Why all the references?

If you look at the pseudo-tree above and think “whats with all the references? Why not simply have X where it says X?”, then you probably dont get the big picture yet. There are many different value types a parser/compiler has to deal with. For instance, there are local variables for a procedure, but there are also class-global fields. Some fields are visible only to the member of that class (public or protected) while others are out of sight for decendants. Then there are unit variables which can be accessed only by unit procedures (both classes and non objective procedures) – depending on if they are defined above or below the implementation marker. Properties on the other hand cannot be handled in the same way, since they involve a getter/setter mechanism.

The only way to deal with all these different types from a unified codebase, is to derive them all from a common value class, which implements standard methods for getting and setting values. Hence the number “0” or “255” is not explicitly defined, instead these constant values (constant because they dont change) must be registered and evaluated as they are used.

The original values would be replaced when a code generator uses them, resolved back to their initial representation. A C# codegen would end up with something like this:

for (x=0;x<256;;) {
  x++;
}

Writing a new compiler

Parsers and compiler AST models are extremely complex and time-consuming to build, but they are also the most rewarding intellectual endeavor you can pursue in the world of computing. At least I think so. Making a game or a utility program is ofcourse fun, but nothing beats making a real, life compiler which takes code and “makers something” from it.

I really hope this little article inspire more use of Castalia, both the commercial product – as well as it’s free Delphi parser. There is ample room in the world of Delphi for new and interesting compilers, so give it a whirl!

Chainable effects for Smart Mobile Studio

Just made a small but very effective SVN commit to QTXLibrary. The non-callback variations of the effect methods are now functions, allowing for some interesting chainable effects.
For instance, you can now do this:

FPanel1.fxMoveTo(0.4).fxFadeIn(0.4).fxMoveDown(0.3).fxZoomOut(0.2);

Since all the fx–() methods returns the “self” value, this means you can execute effects like this. And they will execute in sync due to the callback checking (if the effect is busy).

All the power of jQuery / jQGUI without all the hassle of raw JS 🙂

Head over to https://code.google.com/p/qtxlibrary/ and update your SVN repo now!

 

Smart Mobile Studio goes native

SMS builder

Yes, you read right. I am presently in the process of completing an add-on for Smart Mobile Studio which turns your compiled JavaScript into native win32 and 64 bit applications. Applications that can be distributed, installed and executed just like ordinary programs.

Multimedia and hybrid application development

With the release of secondary compiler, you will be able to produce native 32-bit and 64-bit hybrid applications, meaning that you take full advantage of the browser’s multimedia capabilities – but also reap the benefits of direct access to the operative system; en-par with Embarcadero Delphi and other native programming languages.

• Create native Windows tile apps
• Perfect for games and multimedia
• Full access to the file-system
• Build web and native apps from the same source with no change
• Removes same-origin policy, giving your apps access to cross-domain data
• Take full advantage of web-workers (threading)

Distribution

The compiler collects and binds all your external resources (JavaScript, html, CSS, images and data files) into a single executable file. The application can either start in full-screen mode (user can exit full screen using the CTRL + ESC key) . The application has no dependencies except for the Chromium Embedded DLL libraries.

Users can expect a significant speed boost as rendering threads are executed with elevated priority, making Smart Mobile Studio a perfect game and multimedia engine.

Alternatives

At the moment you can achieve the same result with Apache Cordova (phonegap) and nodeWebkit. The easiest (command-line) to work with is no doubt nodeWebKit.

JavaScript for the future

Samsung’s latest OS supports JavaScript out of the box

More and more hardware and software supports or runs JavaScript directly. Anything from micro-controllers to the latest mobile operative system from Samsung, Tizen, opens for full application development using nothing but JavaScript. Microsoft likewise surprised everyone when they picked JavaScript as their primary language for Windows Tiles and desktop widgets. And for those situations which demand native code, you simply package your app using Phonegap to achieve just that.

JavaScript is the language which has seen the most growth for the past 4 years – and the language is getting more and more important for future-proof application development.

Firefox JavaScript OS

Smart Mobile Studio leverages JavaScript and makes programming advanced, object-oriented, large-scale applications both fun and enjoyable.  And when I write object-orientation I mean the real deal, not just hyped prototype cloning like other frameworks offer.

For a more in-depth presentation of Smart Mobile Studio click here – or head over to www.smartmobilestudio.com for the full specs.

SMS fonts, once and for all, part 3

Right, in the past two articles (first here and second here) we established two set of routines in the same class. In short, what we have finished is as such:

• The ability to check if a font is installed
• The ability to traverse and find the font used by an html element
• The ability to measure text

Now the last point on that list is extremely tricky. While measuring static, one-liner text (read: text that does not break) is indeed working as expected, perfect for captions and other forms of lengths — more complex segments of html measurement is proving to require more work.

To illustrate what we want to do here:

Display width vs. scroll width

As you can see from this highly scientific diagram i slapped together in paint (the typos are free by the way), by setting the “display” css property to “scroll” (please see css documentation for the display property here), we basically turn the yellow DIV element into a Delphi like TScrollbox. So what we are effectively looking for is clientwidth/clientheight for a TScrollbox which has content more content than is being displayed.

HTML5 is a bit silly regarding this, in that the scrollWidth and scrollHeight properties, which contains the full size of the blue box (the values we want to get) are only there if we have set the display property right — and that the content actually spans beyond the visible display. If not, scrollWidth or scrollHeight may be nil (undefined) depending on what fits within it. This is why i scale the element to 4×4 pixels before measuring – to make sure both values are kosher.

And now for the problem (or feature, depending on how you regard the collected wisdom of the WC3), we have to write to the style for the element, but read back values from the calculated style. Why? Because the world of browsers are not (believe it or not) an exact science. As discussed ad infinitum by now, the browser can assign several styles to an element (hence the “cascading” word in CSS) which results in a final style simply called “the calculated style”. Which the WC3 in their infinite wisdom have decided to store in a completely separate location (document.topframe as opposed to, oh say, element.calculatedStyle).

To make things even more fun, the browser may in fact end up breaking things up to much when we scale down — so truth be told, we may fall back to clientWidth/clientHeight or offsetWidth/offsetHeight instead. Confused yet? Your welcome.

Fixing it

Now, the problem with the old model was simply that by setting the element to 4×4 pixels in size, I was not in fact measuring the actual content – instead I was measuring the largest word in the string. Which gave some pretty funky visual results to be mild.

I have since updated the routine and I am very happy to report that it now works as expected — and finally my casebook demo auto-calculates the height of a new-item perfectly (at least under webkit). I have yet to do battle with Mozilla and Opera, but at least I suspect that Opera will have it’s ducks in a row.

So here is the updated version. I will omit the word “final” because the browser can still throw some quirks at us – but at least now we know what we are dealing with!

Something awesome

QTXLibrary expands the normal TW3CustomControl with attribute storage. I decided to add two methods for measuring text directly on the level of TW3Customcontrol. What it does is grab the font information for that element directly – so you dont have to worry about those parameters, and just provide the text you want to measure. Used the fixed version if you know the width of the target placeholder. Pretty cool!

  TQTXTextMetric  = Record
    tmWidth:  Integer;
    tmHeight: Integer;
    function  toString:String;
  End;

  TQTXFontInfo = Record
    fiName: String;
    fiSize: Integer;
    function  toString:String;
  End;

  TQTXFontDetector = Class(TObject)
  private
    FBaseFonts:     array of string;
    FtestString:    String = "mmmmmmmmmmlli";
    FtestSize:      String = '72px';
    Fh:             THandle;
    Fs:             THandle;
    FdefaultWidth:  Variant;
    FdefaultHeight: Variant;
  public
    function    Detect(aFont:String):Boolean;

    function    MeasureText(aFontInfo:TQTXFontInfo;
                aContent:String):TQTXTextMetric;overload;

    function    MeasureText(aFontInfo:TQTXFontInfo;
                aFixedWidth:Integer;
                aContent:String):TQTXTextMetric;overload;

    function    MeasureText(aFontName:String;aFontSize:Integer;
                aContent:String):TQTXTextMetric;overload;

    function    MeasureText(aFontName:String;aFontSize:Integer;
                aFixedWidth:Integer;
                aContent:String):TQTXTextMetric;overload;

    function    getFontInfo(const aHandle:THandle):TQTXFontInfo;

    Constructor Create;virtual;
  End;

//############################################################################
// TQTXFontInfo
//############################################################################

function TQTXFontInfo.toString:String;
begin
  result:=Format('%s %dpx',[fiName,fiSize]);
end;

//############################################################################
// TQTXFontDetector
//############################################################################

Constructor TQTXFontDetector.Create;
var
  x:  Integer;
begin
  inherited Create;
  FBaseFonts.add('monospace');
  FBaseFonts.add('sans-serif');
  FBaseFonts.add('serif');

  Fh:=browserApi.document.body;

  Fs:=browserApi.document.createElement("span");
  Fs.style.fontSize:=FtestSize;
  Fs.innerHTML := FtestString;
  FDefaultWidth:=TVariant.createObject;
  FDefaultHeight:=TVariant.createObject;

  if FBaseFonts.Count>0 then
  for x:=FBaseFonts.low to FBaseFonts.high do
  begin
    Fs.style.fontFamily := FbaseFonts[x];
    Fh.appendChild(Fs);
    FdefaultWidth[FbaseFonts[x]]  :=  Fs.offsetWidth;
    FdefaultHeight[FbaseFonts[x]] :=  Fs.offsetHeight;
    Fh.removeChild(Fs);
  end;
end;

function TQTXFontDetector.getFontInfo(const aHandle:THandle):TQTXFontInfo;
var
  mName:  String;
  mSize:  Integer;
  mData:  Array of string;
  x:  Integer;
Begin
  result.fiSize:=-1;
  if aHandle.valid then
  begin
    mName:=w3_getStyleAsStr(aHandle,'font-family');
    mSize:=w3_getStyleAsInt(aHandle,'font-size');

    if length(mName)>0 then
    begin
      asm
        @mData = (@mName).split(",");
      end;
      if mData.Length>0 then
      Begin
        for x:=mData.low to mData.high do
        begin
          if Detect(mData[x]) then
          begin
            result.fiName:=mData[x];
            result.fiSize:=mSize;
            break;
          end;
        end;
      end;
    end;
  end;
end;

function TQTXFontDetector.Detect(aFont:String):Boolean;
var
  x:  Integer;
Begin
  aFont:=trim(aFont);
  if aFont.Length>0 then
  Begin
    if FBaseFonts.Count>0 then
    for x:=FBaseFonts.low to FBaseFonts.high do
    begin
      Fs.style.fontFamily:=aFont + ',' + FbaseFonts[x];
      Fh.appendChild(Fs);
      result:= (Fs.offsetWidth  <> FdefaultWidth[FBaseFonts[x]])
          and  (Fs.offsetHeight <> FdefaultHeight[FBaseFonts[x]]);
      Fh.removeChild(Fs);
      if result then
      break;
    end;
  end;
end;

function TQTXFontDetector.MeasureText(aFontInfo:TQTXFontInfo;
         aFixedWidth:Integer;
         aContent:String):TQTXTextMetric;
Begin
  result:=MeasureText(aFontInfo.fiName,aFontInfo.fiSize,aFixedWidth,aContent);
end;

function TQTXFontDetector.MeasureText(aFontInfo:TQTXFontInfo;
         aContent:String):TQTXTextMetric;
Begin
  result:=MeasureText(aFontInfo.fiName,aFontInfo.fiSize,aContent);
end;

function TQTXFontDetector.MeasureText(aFontName:String;aFontSize:Integer;
         aContent:String):TQTXTextMetric;
var
  mElement: THandle;
Begin
  if Detect(aFontName) then
  begin
    aContent:=trim(aContent);
    if length(aContent)>0 then
    begin
      mElement:=BrowserAPi.document.createElement("p");
      if (mElement) then
      begin
        mElement.style['font-family']:=aFontName;
        mElement.style['font-size']:=TInteger.toPxStr(aFontSize);
        mElement.style['overflow']:='scroll';

        mElement.style['display']:='inline-block';
        mElement.style['white-space']:='nowrap';

        mElement.innerHTML := aContent;
        Fh.appendChild(mElement);

        result.tmWidth:=mElement.scrollWidth;
        result.tmHeight:=mElement.scrollHeight;
        Fh.removeChild(mElement);

      end;
    end;
  end;
end;

function TQTXFontDetector.MeasureText(aFontName:String;aFontSize:Integer;
         aFixedWidth:Integer;
         aContent:String):TQTXTextMetric;
var
  mElement: THandle;
Begin
  if Detect(aFontName) then
  begin
    aContent:=trim(aContent);
    if length(aContent)>0 then
    begin
      mElement:=BrowserAPi.document.createElement("p");
      if (mElement) then
      begin
        mElement.style['font-family']:=aFontName;
        mElement.style['font-size']:=TInteger.toPxStr(aFontSize);
        mElement.style['overflow']:='scroll';

        mElement.style.maxWidth:=TInteger.toPxStr(aFixedWidth);
        mElement.style.width:=TInteger.toPxStr(aFixedWidth);

        mElement.innerHTML := aContent;
        Fh.appendChild(mElement);

        result.tmWidth:=mElement.scrollWidth;
        result.tmHeight:=mElement.scrollHeight;

        Fh.removeChild(mElement);

      end;
    end;
  end;
end;