TTween library for Delphi now free

I have asked for financial backing while creating libraries that people want and enjoy, and as promised they are released into open-source land afterwards.

HexLicense was open-sourced a while back, and this time it’s TTween library that is going back to the community.

Tweening?

You have probably noticed how mobile phone UI’s have smooth movements? like on iOS when you click “back” the whole display slides smoothly into view; or that elements move, grow and shrink using fancy, accelerated effects?

This type of animation is called tweening. And the TTween Library makes it super easy to do the same for your VCL applications.

Check out this Youtube video to see how you can make your VCL apps scale their controls more smoothly

You can fork the project here: https://bitbucket.org/cipher_diaz/ttween/src/master/

To install the system as ordinary components, just open the “Tweening.dproj” file and install as normal. Remember to add the directory to your libraries path!

Support the cause

If you like my articles and want to see more libraries and techniques, then consider donating to the project here: https://www.paypal.me/quartexNOR

Those that donate $50 or more automatically get access to the Quartex Web OS repositories, including full access to the QTX replacement RTL (for DWScript and Smart Mobile Studio).

Thank you for your support, projects like Amibian.js and the Quartex Web OS would not exist without my backers!

Building a Delphi Database engine, part four

This article is over six months late (gasp!). Work at Embarcadero have been extremely time consuming, and my free time has been bound up in my ex-patreon project. So that’s why I was unable to finish in a more predictable fashion.

But better late than never — and we have finally reached one of the more exciting steps in the evolution of our database engine design, namely the place where we link our metadata to actual data.

So far we have been busy with the underlying mechanisms, how to split up larger pieces of data, how to collect these pieces and re-assemble them, how to grow and scale the database file and so on.

We ended our last article with a working persistence layer, meaning that the codebase is now able to write the metadata to itself, read it back when you open the database, persist sequences (records) – and our humble API is now rich enough to handle tasks like scaling. At the present we only support growth, but we can add file compacting later.

Tables and records

In our last article’s code, the metadata exposed a Table class. This table-class in turn exposed an interface to our field-definitions, so that we have a way to define how a table should look before we create the database.

You have probably taken a look at the code (I hope so, or much of this won’t make much sense) and noticed that the record class (TDbLibRecord) is used both as a blueprint for a table (field definitions), as well as the actual class that holds the values.

If you look at the class again (TDbLibRecord can be found in the file dblib.records.pas), you will notice that it has a series of interfaces attached to it:

• IDbLibFields
• IStreamPersist

The first one, which we expose in our Table as the FieldDefs property, simply exposes functions for adding and working with the fields. While somewhat different from Delphi’s traditional TFieldDefinition class, it’s familiar enough. I don’t think anyone who has used Delphi with databases would be confused around it’s members:

  IDbLibFields = interface
    ['{0D6A9FE2-24D2-42AE-A343-E65F18409FA2}']
    function    IndexOf(FieldName: string):  integer;
    function    ObjectOf(FieldName: string): TDbLibRecordField;

    function    Add(const FieldName: string; const FieldClass: TDbLibRecordFieldClass): TDbLibRecordField;
    function    Addinteger(const FieldName: string): TDbLibFieldInteger;
    function    AddStr(const FieldName: string): TDbLibFieldString;
    function    Addbyte(const FieldName: string): TDbLibFieldbyte;
    function    AddBool(const FieldName: string): TDbLibFieldboolean;
    function    AddCurrency(const FieldName: string): TDbLibFieldCurrency;
    function    AddData(const FieldName: string): TDbLibFieldData;
    function    AddDateTime(const FieldName: string): TDbLibFieldDateTime;
    function    AddDouble(const FieldName: string): TDbLibFieldDouble;
    function    AddGUID(const FieldName: string):  TDbLibFieldGUID;
    function    AddInt64(const FieldName: string): TDbLibFieldInt64;
    function    AddLong(const FieldName: string): TDbLibFieldLong;
  end;

But, as you can see, this interface is just a small part of what the class is actually about. The class can indeed hold a list of fields, each with its own datatype – but it can also persist these fields to a stream and read them back again. You can also read and write a value to each field. So it is, for all means and purposes, a single record in class form.

The term people use for this type of class is: property bag, and it was a part of the Microsoft standard components (Active X / COM) for ages. Its probably still there, but I prefer my own take on the system.

In this article we are going to finish that work, namely the ability to define a table, create a database based on the metadata, insert a new record, read records, and push the resulting binary data to the database file. And since the persistency is already in place, opening the database and reading the record back is pretty straight forward.

So this is where the metadata stops being just a blue-print, and becomes something tangible and real.

Who owns what?

Before we continue, we have to stop and think about ownership. Right now the database file persists a global list of sequences. The database class itself has no interest in who owns each sequence, if a sequence belongs to a table, if it contains a picture, a number or whatever the content might be — it simply keeps track of where each sequence begins.

So the first order of the day is to expand the metadata for tables to manage whatever records belongs to that table. In short, the database class will focus on data within its scope, and the table instances will maintain their own overview.

So the metadata suddenly need to save a list of longwords with each table. You might say that this is wasteful, that the list maintained by the database should be eliminated and that each table should keep track of it’s own data. And while that is tempting to do, there is also something to be said about maintenance. Being able to deal with persisted data without getting involved with the nitty-gritty of tables is going to be useful when things like database compacting enters at the end of our tutorial.

Locking mechanism

Delphi has a very user-friendly locking mechanism when it comes to databases. A table or dataset is either in read, edit or insert mode – and various functions are allowed or prohibited depending on that state. And it would probably be wise to merge the engine with Delphi’s own TDatabase and TTable at some point – but right now im more interested in keeping things clean and simple.

When I write “locking mechanism” I am not referring to a file-lock, or memory lock. Had we used memory-mapped files the locking mechanism would have been more elaborate. What I mean with a lock, is basically placing a table in one of the states I mentioned above. The table needs to know what exactly you want to do. Are you adding a record? Are you editing an existing record? The table code needs to know this to safely bring you from one mode to the next.

Suddenly, you realize why each table needs that extra list, because how is the table going to allow methods like first, next, last and previous? The record-list dealt with by the database is just a generic, non-ordered ledger of sequences (a global scope list if you will). Are you going to read all records back when you open the database to figure out who owns what?

A call to First() will mean a completely different offset for each table. And the logical way to handle this, is to give each table it’s own cursor. A class that keeps track of what records belongs to the table, and also keeps track of whatever states the table is in.

The database cursor

Since we are not up against Oracle or MSSQL here, but exploring database theory, I have kept the cursor as simple as I possibly could. It is a humble class that looks like this:

The idea of-course is that the table defaults to “read” mode, meaning that you can navigate around, record by record, or jump to a specific record using the traditional RecNo property.

The moment you want to insert or edit a record, you call the Lock() method, passing along the locking you need (edit or insert). You can then either cancel the operation or call post() to push the data down to the file.

The Lock() method is a function (bool), making it easier to write code, as such:

  if Database.Table.Cursor.Lock(cmInsert) then
  begin
    with Database.GetTableByName('access_log').cursor do
    begin
      Fields.WriteInt('id', FUserId);
      Fields.WriteStr('name', FuserName);
      Fields.WriteDateTime('access', Now);
      Post();
    end;
  end else
  raise exception.create('failed to insert record');

Im sure the are better designs, and the classes and layout can absolutely be made better; but for our purposes it should be more than adequate.

Reloading record data

In the previous articles we focused on writing data. Basically taking a stream or a buffer, breaking it into pages, and then storing the pages (or blocks) around the file where there was available space.

We cleverly crafted the blocks so that they would contain the offset to the next block in a sequence, making it possible to read back a whole sequence of blocks by just knowing the first one (!)

A part of what the cursor does is also to read data back. Whenever the RecNo field changes, meaning that you are moving around the table-records using the typical Next(), Previous(), First() etc functions — if the cursor is in read mode (meaning: you are not inserting data, nor are you editing an existing record), you have to read the record into memory. Otherwise the in-memory fields wont contain the data for that record.

Creating a cursor

One note before you dive into the code: You have to create a cursor before you can use it! So just creating a table etc wont be enough. Here is how you go about doing this:

Creating the cursor will be neatly tucked into a function for the table instance, we still have other issues to deal with.

What to expect next?

Next time we will be looking at editing a record, commiting changes and deleting records. And with that in place we have finally reached the point where we can add more elaborate functionality, starting with expression parsing and filters!

You can check out the code here: https://bitbucket.org/cipher_diaz/dbproject/src/master/

Support the cause

If you like my articles and want to see more libraries and techniques, then consider donating to the project here: https://www.paypal.me/quartexNOR

Those that donate $50 or more automatically get access to the Quartex Web OS repositories, including full access to the QTX replacement RTL (for DWScript and Smart Mobile Studio).

Thank you for your support, projects like Amibian.js and the Quartex Web OS would not exist without my backers!

/Jon

Amiga Disrupt: talk from the heart

My previous article regarding the dreadful state the Amiga Kernel and OS finds itself in, primarily perpetuated by Italian company Cloanto, must have hit a nerve. My mailbox has been practically bombarded by people who are outraged by Cloanto (and Hyperion has got a fair bit of blame too). And indeed, there were errors made in that article (more about that below).

What I find strange, if not borderline insane, is how ingrained people are to their company or “team”. I have never understood people who watch soccer, who get physically upset over a game – or who demonstrate complete and utter loyalty to a team no matter how ridiculous that team might be. To me,  soccer is just 22 grown men running around in their underwear chasing an inflated dead animal.

Thankfully, “Amiga hooligans” are a minority in the community. And it doesn’t really matter what topic you bring to the table, because they will oppose it either way. It’s what they do. The majority of the community are grown men and women with families, jobs and a life that has nothing to do with shared memories of the Commodore Amiga. And despite our differences we have one thing in common: a desire to see the system we grew up with flourish; a system that never failed and that despite its age has features and mechanisms that modern system lacks. It was management that failed, not the product.

As a developer, having to watch the brilliance of Amiga OS “rot on the wine” as the saying goes, is heartbreaking. The potential in the OS, even if we were to do a clean re-write, is astronomical. The ease of use alone for education, or as a low-cost alternative to Linux on embedded systems, has practical value far beyond gaming; which tragically is the only thing some people associate the technology with.

Points of view

The initial point of my article was not to paint Cloanto as the villain and Hyperion as the hero. I think everyone that has kept an eye on the Commodore saga and aftermath knows full well that none of the companies, both present and past, are without flaw. People don’t start companies for fun, but to do business. And the moment money is involved – human beings can demonstrate both excellence and selfishness. It’s human to make mistakes, and what ultimately matters is how we deal with them.

It all boils down to vantage-point. If your only ambition is to play some retro-games, then you will no doubt be happy with Cloanto’s Amiga Forever. If you enjoy software development and have coding as a hobby, then a full UAE setup, including cross compilers and real hardware will more than cover your needs.

So from those points of view, where you have already parked Amiga OS in the past as a dead system and hobby, I fully understand that you don’t care who did what, or the motives behind various strategic moves. Nothing wrong with that, people are different.

But what both those viewpoints have in common is that they are looking backwards to the past, rather than forward to a potential future. If you recognize that, and you yourself look to the future, then your expectations will be higher. You will care about how the IP is maintained, and also how the legacy is cared for. Legally it’s ultimately nobody’s business what Hyperion or Cloanto does with their intellectual property, but they have to remember that they are responsible for a computer legacy stretching back to the very beginning of home computers.

David’s book about what went on inside Commodore is quite a wake-up call. Go buy it ASAP!

The reason people refuse to throw Amiga out after so many years, is because the product was cut down before it’s time. Some compare it to the Betamax tragedy, where VHS despite being a lesser product ended up as the standard. And just like with the Amiga, it was not the product that was the determining factor in the tragedy, it was the lesser qualities of human beings. VHS allowed porn to be shipped en-mass on their format, while Betamax stuck to their principles and family values.

Commodore was thankfully not involved in anything as base, but if you take the time to read David Pleasance’s book: Commodore the inside story; you will discover that there were some monumental mistakes made in the name of, shall we say, “the lesser instincts of man“?. If you havent read his book then please do, then spend a few hours finding your jaw on the floor. It is absolutely shocking what went on behind closed doors in the company.

Mistakes in my post

The source of the mistake I wrote about, namely that of Acer’s ownership, is rooted in a simple misunderstanding. My focus was initially not on the ownership of the Amiga alone, but rather where has the Commodore patent portfolio gone? Commodore had been in business since 1954, and entered the computer market in 1979 with a MOS 6504 powered chess machine. A company with the level of growth and production over so many decades must have racked up some valuable patents, be they mechanical or electronic. I have never met Jack Trammell in person, but with regards to what I have read about the man, he would not miss an opportunity to make money or be whimsical about patents. So where did it all go?

Prior to my talk with Trevor Dickinson, I looked around to see who ended up with said portfolio (the proverbial needle in a haystack), I talked to several individuals in the community about this, googled, read articles  – and was left with 3 potential candidates: HP, Acer and Asus.

While searching I came across the following video, and the ingress underlines Acer as the patent owner:

Acer is again mentioned as owning patents

When I then had a quick chat with Trevor and the name Acer turned up a third time, I saw no reason to question this. It was ultimately not the point of my post anyway.

The next question was to determine the relationship between said owner and those running the Amiga side of things (Cloanto and Hyperion). There were two logical possibilities: either these companies owned, in the true sense of the word, different parts of the legacy — or they functioned under a branding franchise. Meaning that they have been granted the right to evolve, sell and/or represent the Amiga name and technology with obligations of royalties. This is a pretty common business model, IBM being the archetypical example, so it would not be uncommon.

And that is ultimately the mistake. In retrospect I should have known there was no large company involved, because a stable corporation would never have allowed their IP to be mangled and dragged through the gutter like the Amiga have endured.

Having said that, it doesn’t really change much. I got an email saying that Cloanto have indeed given the authors of UAE money, which I hope is true because without the developers of UAE, the Amiga community would be abysmal. They have done 90% of the lifting, yet receive little praise for their work. But again – I was unable to find anything online where this could be confirmed.

It has also been stated that Amiga Inc was both tricked, abused and bullied by Hyperion. Yet the escapades of Amiga-Inc seem to have vanished into thin air:

“later that year, Amiga Inc. used some sleight of hand to escape a pending bankruptcy. Amiga sold its assets to a shell company called KMOS—a Delaware firm headquartered in New York—then renamed KMOS back to Amiga Inc. It tried to use these shenanigans to get out of the clause in its contract with Hyperion that would revert ownership of OS 4 if Amiga Inc. ever went under. Then, to top it off, Amiga sued Hyperion for not delivering OS 4 on time and demanded the return of all source code.” –Source: Ars Technica

Oh and then there was the “death threat” email. Where my post was said to be so diabolically crafted, so insiduius and evil – that i was responsible for possible death threats. I don’t even know how to respond to that, because the poo-nami that Cloanto is experiencing is the result of 15 years of silence; where the only communication has been to threaten Amiga users who accidentally shared a 512kb rom-file from the late bronze age with legal action. I think you gravely over-estimate my influence in the matter.

Right now Cloanto seem to run around pretending to be Santa. With promises of open-source and a future for their Amiga OS 4.1 (yes you read right) and that 3.1.4 is also theirs. First of all, Hyperion got that source-code as a part of the settlement with Amiga Inc (the quote above from ARS-Technica demonstrates how Amiga Inc treated Hyperion, not the other way around).

From a video posted by the 10 minute amiga retro-cast

Secondly, the Amiga OS 3.x source code has been available on the pirate bay for 4 years now? So if Cloanto indeed are so secure in their role as rightful heir to the Amiga throne, they can open source the code in a matter of hours. Just download, slap a GPL license on the files and push it out.

To nullify a 15-year-old settlement bound by contract, which is what must happen for them to have rights to their claims — that is something I wont hold my breath waiting for.

A viable business model

After my initial post people have dragged poor Trevor Dickinson into the debate, complaining to him about statements made by me. That is unfortunate because Trevor is not involved in our opinions at all. He even corrected me about mistakes I made in the previous article – and have absolutely not been a catalyst (quite the opposite!).

The Amiga history after the Commodore era is so convoluted, that his article series on the subject ended up spanning 12 issues of AF Magazine (!) Compare that to my two page brain fart. I also underlined that I had left out most of the details because rehashing the same tragedy over and over is paramount to explaining Game Of Thrones backwards in Sanskrit.

If we push all the details and who said what to the side for a moment, and look at the paths we have – it begins with a simple choice: you can look to the past and stick to “retro” computing. If that is the case then you will have no interest in anything I have to say, and that is fine. High five and enjoy.

If you look to the future, then suddenly we have some options before us: you have FPGA, like the FPGA-Arcade, the Vampire, MISTer and other, similar FPGA based systems. They have one thing in common and that is the 680×0 CPU.

Then you have software emulation, WinUAE being the trend-setter and various forks like UAE4Arm, FS-UAE and so on. This is perhaps the most versatile solution since it can do things difficult to achieve under real hardware.

Then we have the next generation and re-implementations. This is where Aros and it’s variations (AEROS, ARES et-al), Amiga OS 4.x and Morphos comes in.

I can’t see that we even need the legacy systems for much longer

And last but not least, cloud implementations like Amibian.js.

But in order for there to be any future where the core technology can grow, the technology has to serve a function in 2019. It doesn’t matter if the IPC layer is awesome, or that Amiga OS had REXX support 20 years before Mac OS. A modern system have to give users in this decade a benefit — otherwise there is no business model to talk about. And that is also my point. If we exclude web tech for now and look at the different paths, only two of them have the potential to deliver modern and unique functionality; and in my view that is Amiga OS 4 and Morphos.

FPGA will disrupt everything at some point

Vampire could perform a miracle and optimize their 68k architecture to the point where it can serve as a good embedded system, but even if possible, they are still held back by their dependency on classic Amiga OS. A partnership between Hyperion and Apollo would indeed be interesting, who knows. Although I would love to see the Apollo team fork Aros and shape that into what it could become with a bit of work.

Morphos is rumored to be moving their codebase to x86. This is just a rumour and I havent seen any documentation around that. If this is true then I feel it is a mistake, because NVidia and roughly 100 other major players are about to attack Intel on all fronts with RISC-V – and ARM is set to replace x86 in consumer electronics faster than most expected. Apple just announced that ARM based laptops are in the making.

I should add that this is also why I decided to write Amibian.js using web technology, because regardless of which CPU or architecture that becomes dominant in the next decade, web tech will always be there. So it allows us to abstract away the costly dependency on hardware, and instead focus on functionality.

PPC for the win?

In an interesting twist of fate, PPC could actually come out far better than anticipated – but not in the way you might think. Work is being done to make PPC a first class FPGA citizen. FPGA is fantastic in many ways, but it’s the intrinsic abillity to “become” whatever technology you describe that is revolutionary.

While it’s still in its infancy, the potential is there to render instruction-sets and architectures a preference rather than a requirement. If anything, the Vampire IV is a demonstration of just that.

So code currently bound to PPC could use FPGA as an intermediate solution while the codebase is ported to more viable platforms.

So whats the problem?

The next question then becomes: what exactly is stopping the owners from moving forward? Why dont the companies that hold the various IP’s roam silicon-valley in search of funding? And it’s here that we face the situation I briefly painted a picture of in my last post: they are in a perpetual stale-mate.

And in my view (as a developer looking forward) Cloanto, whose primary focus is to provide for the legacy market, is constantly getting in the way of Hyperion – which is looking at the future. As far as innovation and managing the legacy of Commodore is concerned, Cloanto has been asleep at the wheel for over a decade. They only woke up when it could cash-in on its C64 assets. I have no number as to how many c64 mini’s have been sold around the world, but its been a massive success. And it would be foolish to think that they have no plans to repeat the success with an Amiga model — effectively hammering the final nail in the coffin. After that, the Amiga is forever a legacy system.

Well. This case is already boring the hell out of me, so I will just leave them to it.

But looking at the various paths forward, from where I stand Hyperion and OS 4.x is the only viable business model. Providing the goal is to bring the technology back into the consumer-market and evolve the technology as an alternative to Windows, OS X and Linux. If the goal is just milk the system one final time, then I would say they are already there.

I honestly could not care less at this point. They have been asleep for so long, that they have become irrelevant. The future is in cloud, clustering and hardware abstraction — and Amibian.js is already far more interesting than anything cloanto has on offer.

But make no mistake: If the parties involved dont get their shit together, come 2022 and we will implement a native OS ourselves and open source it through torrents. The Quartex consortium is deadly serious about this. The new QTX is made up of members from various established groups back in the day, now in our 40s and 50s. Like all amiga users we have tolerated this for two decades, but enough is enough. Unlike the average gamer most of us are professional developers with decades of experience.

They have until 2022, if nothing has changed, we will finish this for them

And that was my five cents on that matter, and the last post I will do on this dumpsterfire of a topic.

VMWare: A Delphi developers best friend

Full disclosure: I am not affiliated with any particular virtualization vendor of any sorts. The reason I picked VMWare was because their product was faster when I compared the various solutions. So feel free to replace the word VMWare with whatever virtualization software suits your needs.

On Delphi Developer we get new members and questions about Delphi and C++ builder every day. It’s grown into an awesome community where we help each other, do business, find jobs and even become personal friends.

A part of what we do in our community, is to tip each other about cool stuff. It doesn’t have to be directly bound to Delphi or code either; people have posted open source graphic programs, video editing, database designers – as long as its open source or freeware its a great thing (we have a strict policy of no piracy or illegal copying).

Today we got talking about VMWare and how its a great time saver. So here goes:

Virtualization

Virtualization is, simply put, a form of emulation. Back in the mid 90s emulators became hugely popular because for the first time in history – we had CPU’s powerful enough to emulate other computers at full speed. This was radical because up until that point, you needed special hardware to do that. You had also been limited to emulating legacy systems with no practical business value.

VmWare Workstation is an amazing piece of engineering

Emulation has always been there, even back in the 80s with 16 bit computers. But while it was technically possible, it was more a curiosity than something an office environment would benefit from (unless you used expensive compute boards). We had to wait until the late 90s to see commercial-grade x86 emulation hitting the market, with Virtuozzo releasing Parallels in 1997 and VMWare showing up around 1998. Both of these companies grew out of the data-center culture and academia.

It’s also worth noting that modern CPU’s now support virtualization on  hardware level, so when you are “virtualizing” Windows the machine code is not interpreted or JIT compiled – it runs on the same CPU as your real system.

Why does it matter

Virtualization is not just for data-centers and server-farms, it’s also for desktop use. My personal choice was VMWare because I felt their product performed better than the others. But in all fairness it’s been a few years since I compared between systems, so that might be different today.

A screengrab of my desktop, here showing 3 virtual machines running. I have 64 gigabyte memory and these 3 virtual machines consume around 24 gigabytes and uses 17% of the Intel i7 CPU power during compile. It hardly registers on the CPU stats when idle.

VMWare Workstation is a desktop application available for Windows, Linux and OS X. And it allows me to create virtual machines, or “emulations” if you like. The result is that I can run multiple instances of Windows on a single PC. The virtual machines are all sandbox in large hard-disk files, and you have to install Windows or Linux into these virtual systems.

The bonus though is fantastic. Once you have installed an operating-system, you can copy it, move it, do partial cloning (only changes are isolated in new sandboxes) and much, much more. The cloning functionality is incredibly powerful, especially for a developer.

It also gives you something called snap-shot support. A snapshot is, like the word hints to, a copy of whatever state your virtual-machine is in at that point in time. This is a wonderful feature if you remember to use it properly. I try to take snapshots before I install anything, be it larger systems like Delphi, or just utility applications I download. Should something go wrong with the tools your work depends on — you can just roll back to a previous snapshot (!)

A great time saver

Updates to development tools are always awesome, but there are times when things can go wrong. But if you remember to take a snapshot before you install a program, or before you install a component package — should something go wrong, then rolling back to a clean point is reduced to a mouse click.

I mean, imagine you update your development tools right? Suddenly you realize that a component package your software depends on doesn’t work. If you have installed your devtools directly on the metal, you suddenly have a lot of time-consuming work to do:

• Re-install your older devtools
• Re-install your components and fix broken paths

That wont be a problem if you only have 2-3 packages, but I have hundreds of components install on my rig. Just getting my components working can take almost a full work-day, and I’m not exaggerating (!).

With VMWare, I just roll back to when all was fine, and go about my work like nothing happened.

I made a quick, slapdash video to demonstrate how easy VmWare makes my Delphi and JS development. If you are not using virtualization I hope this video at least makes it a bit clearer why so many do.

Click the image to watch the video on YouTube