Archive

Archive for June, 2017

Freepascal online, awesome work!

June 20, 2017 Leave a comment

Freepascal is, with the exception of gcc – the gnu c/c++ compiler toolchain, probably the most versatile and adaptable compiler in the world. Unlike most compilers outside the C camp – Freepascal is complete toolchain, and it has more than a few similarities to c/c++ in that it’s largely self-reliant.

MUI_PascalDemo

Freepascal scales from the old mc68000 family to the latest Intel i7, impressive!

FPC is easy to port over to new (or old) platforms, and as far as languages go – object pascal has a level of productivity that I have yet to find in other languages.

Now I have no interest in language wars or fanboy stereotyping, all languages have aspects that are unique and beneficial. So this is not an attempt to promote A over B.

Targeting both old and new from a single codebase

While mainstream and work related topics never brush into this – there is actually a healthy group of people still using older platforms. Naturally this is well within the “hobby” range – where passion and curiosity are the prime motivators rather than cash. But still, it is interesting to see how little of modern software is capable of scaling back to processors 10, 15 or 20 years out of date.

Alb42 is the nickname of a very industrious individual who has taken it on himself to port freepascal back to his roots. Which just happens to be the Amiga line of computers.

MiniSlugAmiga

Some of the best arcade games ever made are “vintage” by nature

But he has also done a lot of work for modern systems. A new Amiga was actually released this year (although most people havent noticed it) – and finding development software for a brand new platform is not easy. You have a variant of C/C++ naturally – but the honest truth is that C/C++ is not the language most people use to create desktop productivity software in. That is left to object pascal, C# and variations of Java. Delphi alone is responsible for a huge portfolio of popular software on Windows for instance. Only surpassed recently by monoliths like C#.

So getting freepascal running on Amiga OS 4 (which is the latest and modern operating system shipping with the new Amiga x5000 model) is very important to the market share of the platform.

Do it online!

Alb42 has setup an online compiler testbed where you can type in your freepascal snippets and compile to executable code on the spot. You can choose between MorphOS, Classic 68k, Aros (x86) and Amiga OS 4 (ppc). It’s a very simple setup yet it’s possible to do ad-hoc compilation and play around with the system ūüôā

onlinecompile

While humble, Alb42 has setup an interesting online kitchen sink that demonstrates how flexible and powerful object pascal really is

To give it a go, head over to: https://home.alb42.de/fpamiga/

And as always, visit his blog for both VMware images with cross compilation and binaries for your favorite retro system.

But perhaps even more important, your new x5000 (!)

Dont forget to join us at Amiga Dispatch on Facebook.

Cheers!

A day with the Tinkerboard, part 2

June 10, 2017 Leave a comment

Please read the first article here before you continue.

Having gone through a wealth of alternative boards, alternative to the Raspberry PI 3b that is; I have to admit that I am really, really disappointed by the results. Not just with the Tinkerboard but also boards like the ODroid XU4 and the whole fruit-basket of banana, orange and pineapple PI clones.

asus

What a waste

And please keep in mind that I have not been out to discredit anyone. I have been genuinely interested in these alternatives for many reasons, not just emulation or multimedia. I have no problem paying that extra 15£ for a system that delivers twice the power of the PI, in fact I even paid close to 150£ for the UP board. Which is also the only board that so far has delivered on its promise. Microsoft Windows and the horrors of eMMC storage considered.

Part of what I work with these days include node.js and full screen browser views, often touch enabled for POS (point of sale) type machines. This is an exciting field because for the first time in history we can now create rich, responsive and rock solid solutions using off the shelves web technology.

So while emulation of older systems is interesting for hobby purposes, the performance of vital technologies like the browser – and consequently the performance of JavaScript and the underlying JIT compilation, is naturally paramount for what I do.

So I was really looking forward to the extra power these boards advertize. Both CPU, number of cores, extra memory and last but not least – the benefits of a fast GPU. The latter directly impacts the effects we can use in our user-interface. And while effects may seem superficial, its important for touch feedback and the customers experience.

desktop_embedded

We have all come across touch devices that give little or no visual feedback. We all know what its like when you type something and the UI just hangs while talking with a server. This is why web tech is so important today. In many ways Html5 and websocket is re-defining what we think of as software.

So while I bring little pie-charts and numbers to the table, I do bring honesty and perspective. It’s not all fun and games.

Was it really fair?

It has been voiced by a couple that perhaps my way of testing things is unfair. Like I mentioned in my previous article the PI has a healthy head-start and a rich community that not only consumes and demands, but also contributes substantially to the value of the product. A well established community around a product is worth its weight in gold. The alternative boards have yet to establish that and when it comes to the Tinkerboard, it’s pretty much just getting started.

Secondly, as far as emulation goes, we have to remember that those who make these products rarely have emulation in mind (and probably not Amiga which by any standard is esoteric compared to Nintendo and Sega consoles). If we look at what people use gadgets like the Raspberry PI for, I think products like Kodi, Apache and various DIY programming experiments out-rank all the retro systems combined. And it’s clear that the people behind Tinkerboard is more media oriented than anything else; it ships with hardware support for 4k video playback after all.

Having said that, I do feel I gave both the ODroid and Tinkerboard a fair trial. I did not approach these boards as a developer, but rather as a consumer. I have focused on the experience of familiar things – like using a browser and how well JavaScript runs, does the UI lag under stress; things that any customer will face after purchase.

UAE4Arm vs FS-UAE

As far as UAE (Unix Amiga Emulator) there is one thing that might be unfair. Amibian is based around something called UAE4Arm which is a highly optimized version of the Amiga emulator. I would even go so far as to say its THE most optimized UAE version on any platform – Windows and OSX included.

The Amiga was a lot more than just a games machine, it was way ahead of both Windows and OS X for a decade

The Amiga was a lot more than just a games machine, it was way ahead of both Windows and OS X for a decade

Comparing that with FS-UAE is perhaps unfair. FS-UAE is based on the older, non optimized codebase. It’s known for stable emulation and good JIT performance. Sadly the JIT engine only works on x86 and is thus disabled on ARM systems. So to be honest it can’t hold a candle against UAE4Arm. The latter uses lookup tables for everything, fast switch cases instead of long sequences of “if then else” code (easier for the compiler to optimize), and it benefits from hardcore llvm level 3 optimization. So I agree, comparing FS to UAE4Arm is like comparing a tiger with a kitten. It’s not even in the same ballpark.

Optimization on this level is almost a lost art these days. People are used to languages like C# and Java – which are hopelessly bloated (for those that have seen what raw assembler and hand optimized pascal can deliver). On the Amiga we learned how to shave off a clock-cycle here and there – and the end result really made a huge difference.

The same type of low-level, every cycle counts, type optimization is used throughout UAE4Arm. FS-Uae is safe, slow and can hardly deliver A500 speed on a Raspberry PI. Yet UAE4Arm spits out 3.2 times the power of an Amiga 4000 with a 68040 cpu. That was the flagship power machine back in the day, and this little $40 card delivers 3 times the power under emulation.

Its like comparing a muffled 1978 Lada with a pissed off 2017 Lamborghini.

Fair is fair

Having tested just about every distro I could find for the Tinker, even back-tracking to older releases in hope that the drivers there would be better suited for the board (perhaps the new drivers were released in a hurry, or a mistake was made) the last straw really was Android. My hope was that Android could have better drivers since it sees a lot of development. It has giants like Google and Samsung behind it – so the last glimmer of light was that Asus had focused on Android rather than Linux.

podcast-event

Will Android save the say and give the Tinker it’s due credit?

I downloaded the latest image, burnt it and booted – thinking that this should finally put things right. Well it didn’t. Quite the opposite.

The first boot took almost an hour. Yes, an hour. I suspect this is because it did a network restore – downloading packages and requirements from the Internet. But that is still no excuse. I have tried Android on the PI it and it was up and running in less than 10 minutes; and that includes having to set some initial user information.

When the Android desktop finally came into view, it was stripped down with no repository package manager. In other words you would have to manually install packages. But that would have been OK had it not been for the utterly sluggish performance.

Android on the PI is no race-car, but at least it doesn’t freeze and lock-up every five second. Clicking on a link in the browser froze the whole system, and you were asked if you wanted to terminate the process again and again while Android figured out the meaning of life or whatever. Every single time.

I’m sorry but this is the worst experience I have ever had with a board. The ODroid XU4 at least tries to deliver a good experience. I had no problems finding Linux editions that ran well (especially gaming systems, Kodi and some homebrew distros) – and the chrome build for the ODroid gave far better results than both the PI and Tinkerboard. It was not the power-house it had been hyped up to be, but at least it delivered a normal desktop experience (even the PI locks up when facing demanding tasks, but the Tinker has an 8 core cpu, twice the ram and a supposedly superior GPU. With the capacity of running 8 threads and doing task scheduling in batches of 8 – the Tinkerboard should remain both stable and interactive under pressure).

I am sorry but I cannot recommend the Tinkerboard at all. The ODroid gives you a slight edge over the PI and if someone baked a more optimized Linux distro for it (it ships with a variant of Ubuntu) then ODroid XU4  would kick serious ass. But the Tinkerboard is a complete disaster.

Do not waste your money on the Tinkerboard. It may have the hardware but the software to utilize and express that power is simply not there! It has been the most pointless purchase I have done this year.

A common waste

The common denominator between these alternatives is the notion that more power equals more stuff. Which ultimately results in biting over more than they can chew. A distro like Pixel is lightweight, optimized and the PI foundation has spent a considerable amount of time making sure the drivers run as fast as possible.

Weird Science, a fun but completely fictional movie from 1985. I just posted this to liven up an otherwise boring trip down memory lane

Saying it doesnt make it so, except in Weird Science

What these alternative hardware suppliers don’t seem to get is that – what is the point of having a cpu that is twice as fast as the PI, when you waste it on a more demanding version of Linux? Both the ODroid and Tinkerboard ship with Ubuntu clones. And while Ubuntu is wonderful to work with on a high-end x86 machine, 32 gigabytes of ram and a i7 CPU growling away under your desk, it is without a doubt the worst possible OS you could pick for a tiny ARM board.

To put things into perspective: The Tinkerboard has the “theoretical” cpu power of a Samsung Galaxy S4 mobile phone (2012). Yet in their infinite wisdom they decided Ubuntu is what is really going to show off this fart of a device.

It makes absolutely no sense from a consumer’s perspective. As a user you want that extra power for running programs. You want to use the extra power on your stuff. Yet these guys seem to think that wasting all the extra power on infrastructure is going to score points.

So if anyone from the Tinkerboard or ODroid camp reads this, please remember the old saying “If you cannot innovate, emulate”. Make a fast, stable and small Jesse distro like Pixel. Focus on optimization. Compile whatever you can with maximum optimization. Shave off every clock cycle you can. Hand carve the drivers in assembler if that’s what it takes.

Because the sad result is, that right now the Raspberry PI is outperforming you with lesser hardware. There is no point in buying a Tinkerboard because it actually delivers a worse experience. It has bitten off more than it can chew and it’s trying to be something it’s not.

A day with the Tinkerboard, part 1

June 7, 2017 1 comment

Anyone into IOT mini computers, be it as a hobby or professionally – have no doubt heard about the Tinkerboard. The Tinkerboard is an alternative to the Raspberry PI (aren’t they all) produced by hardware giant Asus. Everything from its form-factor (size) to its color coordinated GPIO pins is more or less the same as the PI. What separates the two is that the Tinker gives you double the CPU power, double the ram and pretty much double of everything. If we compare it with the Raspberry PI 3b that is.

But you don’t get double for free. The Tinkerboard retails on Amazon for USD 60 which means something in the ballpark of 55¬£ for those that live in the UK. For the rest of us that live in far-away places like Norway, prices and taxes may vary.

tinker

Before I continue I want to write a few words about pricing. The word “expensive” is one I keep hearing when it comes to the Tinkerboard, but that is extremely unfair considering what you get for your money.

The Raspberry PI 3b is an awesome product and packs a lot of goodies into a credit-card size computer. The same is true for the Tinkerboard, but they have packed even more stuff into the same space! The difference in price is just 15£ Рbut those 15£ gives you the power to emulate demanding platforms like Nintendo Wii, Sega Dreamcast and Playstation (if you are into retro gaming). Platforms the Raspberry PI 3b dont stand a chance emulating (at least not smoothly).

For serious applications all that extra cpu power can be the difference between success and failure. In many ways the Tinker feels a bit like working on a low-end laptop. If you are building a movie server for your living-room the Tinkerboard will ble able to convert and stream movies much smoother and better than the PI. And if you are looking for a device to power your next kiosk system or POS device, again the extra memory and cpu power is going to make all the difference. And with just 15£ between the two models, I cannot imagine why anyone would pick the PI if gaming or movie streaming is on the menu.

Oh and it has built-in wi-fi and bluetooth support! Here are the specs as listed on Amazon:

  • High performance Quad Core ARM SOC 1.8GHz with 2GB of RAM -The Tinker board features the Rock chip RK3288 Soc with Mali – T764 GPU and 2GB of Dual Channel DDR3 memory
  • Non shared GBit LAN, Shielded Wi-Fi with upgradable antenna support
  • Highly compatible PCB & Topology, Tinker board offer extensive compatibility with SBC accessories & chassis
  • HD Audio & HD & UHD Video Support ‚Äď Tinker board supports HD audio 192/24bit audio along with accelerated HD & UHD ( 4K ) video playback support* requires use of Rock chip video player in TinkerOS
  • DIY Friendly Design ‚Äď Tinker board features multiple DIY friendly use features including a color coded GPIO header, silkscreen PCB and color coded pull tabs

That sounds pretty nice doesnt it ūüôā

One does not simply take on the mighty PI

Being an alternative to the Raspberry PI is not as easy as it sounds. You would imagine that it all boils down to the specs right? So if you beef up the cpu, ram and gpu everyone will throw away their PI’s and buy your stuff? If only that was true. In which case the Tinkerboard would be awesome since it outperforms the PI on every level.

But it’s not that simple. There is more to a product than just raw power.

One of the cool things about the Raspberry PI is the community. The knowledge base of the product if you like. In order to build a community and be a success, everything has to click. I mean just look at the third-party eco-system that exists around the PI; all those companies selling add-ons and widgets, sensors and remote controls. The PI itself is quite dull. It’s all that extra stuff that makes it so fun to play with. And then you have things like excellent customer service, a forum where people help each other out, well written documentation — the value of the product suddenly exceeds whatever you paid for it to begin with.

And the Tinkerboard is not alone in trying to get a piece of the action. It competes with boards like the ODroid XU4, the x86 based UP boards, Latte Panda, The Pine boards and a whole fruit-basket of banana, orange and whatever PI clones. All of them trying to get a bite of the phenomenon that is the Raspberry PI.

So Asus got their work cut out for them.

First impressions

Having unpacked the board and plugged in all the cables (which is exactly as dull as it sounds) the first thing you do is look for software. Which naturally should be a one-click operation on their website.

Well, the first thing that annoyed me was exactly that: the Asus website. The Tinkerboard is not something that belongs on a sterile, streamlined corporate site. It belongs on website that should be easy to navigate, with plenty of information that hobbyists and “tinkers” need to get started. The Asus website simply lacks the warmth and friendliness of the PI foundation website. I even had to google around a couple of times just to find the disk images. These should be readily available on the site, on the first page even – to make sure customers can get up and running quickly.

tinkerweb

It’s a little bit clangy and a little bit jammy

The presentation of the board was ok, but you really shouldnt have to use an external search engine to locate essential data for something you just bought. Unlike the PI foundation, Asus seem to have put little effort into what customers get once they have bought the board. As for community I dont even know if there is one.

When you finally manage to find the bloody download page, these are your options (there are 17 downloads in total, counting older OS images, config files and schematics):

  • TinkerOS 1.8
  • TinkerOS 1.6 Beta (my initial download)
  • Android, also beta

And don’t expect a polished Linux distro like Pixel either. You get a clean distro with some typical stuff pre-loaded (python, perl for some reason, and various tidbits) but nothing close to the quality of Pixel.

Note: Why they would list older beta releases in their downloads is beyond me. It only adds to the confusion of a sterile list of items. And customers who just got their boards will be eager to try it out and hence make a mistake like I did. Beta and unstable releases should be in a separate category, while the stable and up-to-date images are listed first.

The second thing that really irritated me was something as simple as adjusting the keyboard layout and setting the locale. These are simply not present in preferences at all. Once again you have to google around until you find the terminal commands to manually set these things, which utterly defeats booting into a desktop environment to begin with. This might seem trivial for someone who knows Linux well, but to a novice it can take hours. Things this fundamental should be available in preferences or at the very least as a script. Like raspi-config on the PI.

But OK. If we look away from these superficial things and focus on the actual board, things are not half bad.

It’s the insides that counts

The first thing you are going to notice is how much faster the system is compared to the PI. When you start an application on the PI that is demanding, like libre-office or Lazarus, the whole system can freeze-up for a few seconds while the CPU go through the roof. I was pleased to find that this is not the case here (at least not as disruptive). You can start a program and the system continues to be responsive while things load.

What made me furious though was the quality of the graphics drivers. Performance wise the Tinkerboard is twice (actually a bit more) as fast as the PI on all fronts – and the GPU is supposed to deliver a vastly superior graphics experience. Yet when I ran Amibian.js (the Smart Pascal JavaScript desktop system) in Chrome – the CSS hardware effects were painfully slow. I even questioned if GPU was used at all (Note: which I later confirmed was the case, so much for downloading a beta image by mistake). I was amazed at the cpu power of the Tinkerboard because despite having to do everything purely with the CPU (moving pixels in memory, allocating device independent bitmaps, rasterizing layers and complex compositions in X) and it was more than usable.

But it sure as hell was not the superior graphics experience I was hoping for.

I also noticed that when moving a normal desktop window (X window that is) rapidly around the screen, it would lag behind and re-trace your steps until it caught up with the mouse pointer. This was very odd since most composition engines would have eliminated these movements as much as possible – and the GPU would do all the work. Again, I blame the drivers and pondered what could possibly make a GPU perform so badly.

gpuwrong

Im supriced it booted at all, nothing is working

So while the system is notably faster than the PI, I get the sense that the drivers and tooling is not really polished. The hardware no doubt got juice but without drivers being carefully written for the board – most of the extra juice you pay for is wasted.

In the end I opened up chrome and typed “chrome://gpu” to see what was going on. And not surprisingly I was correct. It wasnt using the GPU at all and I had wasted hours on something that should have been clearly marked “Lacks hardware acceleration” in bold, italic, underlined big red letters. Like the PI foundation did when they shipped Raspberry PI 2 without GPU drivers. But they promptly delivered and it was ok, because you were informed up front.

 

Back to scratch

After a trip back to google I downloaded the right (or at least working) image, namely the TinkerOS 1.8 stable. It booted up, resized the partition to fill the whole SD card automatically (and other boring bits).

The first thing I did was start chrome and use the “chrome://gpu” to get some statistics. And thankfully most of the items were now working and active.

gpubetter

Thats better, although i ponder why GPU DIB’s are not active

Emulation, the name of the game

While I have serious tests to perform there is no doubt that what I was most eager to play with was UAE (the Unix Amiga Emulator). Like most retro-heads I have a ton of Raspberry PI’s and ODroid’s around the house with the sole purpose of emulating older game or computer systems. And I think everyone knows that my favorite computer in the whole wide world is the Commodore Amiga.

On the PI, Gunnar’s native Amibian distro gives you awesome performance. If you overclock the ram, gpu and cpu using safe numbers – the PI 3b spits out roughly 3.2 times the power of a Mc68040 based Amiga 4000. In other words 3 times the power of a high-end Amiga from the early 90s. That is quite an achievement for a $40 board. You can only imagine my expectations for a board running more than twice as fast as the PI does (and costing that 15¬£ more).

18986661_10154521981875906_472132585_o

Boots fine but you can forget about using it. The PI is 100 times faster than this

One snag though was that UAE4Arm which is the highly optimized version of the Amiga Emulator doesnt exist for the Tinkerboard. In theory it should compile without problems since the latest version uses SDL exclusively. So there is no odd, deprecated UI toolkit like the old UAE4Arm used.

The only thing available was fs-uae. And while that is a very popular emulator on Windows and Mac, it’s sadly not optimized as much as UAE4Arm is. The latter uses lookup tables to the extreme, all if, then, else statements have been replaced by fast switches — and it’s just optimized beyond anything else running on ARM.

The result on the PI in phenomenal, but sadly fs-uae was all I could play with. I will try to compile UAE4Arm on the Tinkerboard later though. But right now im to busy.

Not optimized at all

It is hard to pinpoint exactly where the problem is, but I suspect the older codebase of fs-uae combined with poorly written drivers is what resulted in the absolutely sluggish behavor.

Dont get me wrong, if all you want to do is play some A500 or A1200 games the Tinker is more than capable. But if you want to run the same high-performance desktop as Amibian gives you on the PI – you can pretty much forget it.

I set the system to emulate an A4000, activated the JIT engine and set my desktop to match the display of the desktop. This should run like hell — yet it was slow as a snail. You could literally see the mouse-pointer slowly trying to keep up as I moved it across the desktop.

One interesting thing though. When I ran fs-uae on the first image, the beta image where everything was crap, it actually ran exceptionally well. But here on the so-called stable and “up to date” version, it was useless for anything but clean floppy gaming.

There are many factors involved in this. The drivers sits at the bottom just above the kernel and exposes the hardware so it can be used through a common API. On top of this you have stuff like OpenGL and on top of that again you have SDL which simplifies typical gaming or graphics tasks. This may sound like a long road for code to travel, but it’s actually very fast. If the drivers expose the power of the GPU that is.

I can only conclude at this point that the Tinkerboard has the firepower, I have seen the result of tests performed by others (and tested a retrogaming image) – and you feel the difference almost immediately when you boot the device. But the graphics drivers seems.. crippled somehow. It’s easy to point the finger at Asus and say they have done a terrible job at writing drivers here, but it can also be a bottleneck elsewhere in the system.

This is something Raspberry PI has got so right. Every part of the system has been optimized for the hardware, which makes the PI feel smooth even under a lot of stress. It delegates all the hard graphical stuff to the GPU – and the GPU just deals with it.

Sadly this is not the case with the Tinkerboard so far. And it really is such a shame because the specs speak for themselves.

Android

Next time: We will be giving Android a test and hopefully that will have drivers that are more up to date than whatever we have seen so far..

Smart Pascal: Memory and pointers

June 4, 2017 Leave a comment

Allocating, manipulating and working with memory is an important part of any programming language. While there are many ways of achieving the same result, comparing operations that work directly with the data using pointers – to code that is purely restricted to high-level, fixed datatype arrays is not even a competition. By using pointers you will be able to copy, fill, move and otherwise do things to large chunks of memory that would otherwise be slow and impractical.

Anyone who has ever coded their own game, or working with large blobs in a dataset knows this. I would even argue that you can’t even write a high-speed game in a native environment without pointers (unless you are using one of those game makers, which is not real programming anyways) – and the same can be said about database engines. Could you imagine writing a database engine without using pointers to move data around in memory? I would hope not.

Im not saying that you cant, I’m simply saying it would be a total waste of time not using pointers when the alternative would be 100 times slower.

The web stack and memory

JavaScript¬†is known for many things, but manual memory management is certainly not one of them. That doesn’t mean JavaScript lacks any features for dealing with memory (those days are thankfully behind us), it simply means that your average JavaScript developer havent explored this aspect to the same extent as a C or Object Pascal developer would.¬†These is no shame in this;¬†If JavaScript is all you know then it would be a field you are unconfortable with.

But the JavaScript Virtual Machine and runtime-environment in a modern browser is actually quite good at memory. A lot has happened to JavaScript these past few years and regardless of what engine you use, be it Spidermonkey under Firefox or V8 under Chrome and Safari (and even Microsoft Edge) they have all been advanced with cool new features that object pascal and C/C++ coders are familiar with.

So fact is that a well written JavaScript function can move data around just as fast as your Object Pascal or C++ code. You just need to know the rules. And also get things into a framework that is easier to work with. JavaScript may have the functionality Рbut like all things JavaScript its not pretty and takes some getting used to. Which is why the Smart RTL is so nice to have.

How does memory work under JSVM?

First a few words about JSVM (the JavaScript virtual machine). First of all the runtime and the environment are two different things. If you have ever used a script engine in your Delphi or Lazarus applications, like DWScript, PascalScript or Lua for that matter Рyou will notice that out of the box these engines have no functionality except the fundamental language. They can run functions and create class instances, but the engine itself have no concept of a file or chunk of memory. Not unless you define those features and register the methods that delivers it.

So it is with the JavaScript runtime environment. It is initially empty. Engines like V8 or Spidermonkey have, out of the box, no special functionality at all. It is up to whomever uses the engine (in this case those that make browsers) to write and expose native objects to the engine. Objects that your code can access and consume.

The way memory is handled under Node.js for instance is much faster than how its done in a browser; because node has been optimized for server tasks rather than client tasks. There are also subtle differences between a node.js buffer-object and a browser buffer-object. This is where the strength of the Smart RTL becomes apparent, because it shields you from these differences and gives you a unified API that deals with all of this for you. So no matter if you work with node.js or FireFox – the same code will run identical on both JSVM configurations.

OK, lets look at what JavaScript has to offer. As mentioned above JavaScript is empty by default, and its up to the browser or node to expose a set of common objects you can access and use. And these objects are first and foremost:

  • Buffers
  • Views
  • Blobs

PS: We can look away from the latter since its not really interesting at this point.

Right, as you can probably guess the buffer object is the actual data. Or more correct, it is the container object for your data. It doesn’t have any read or write methods and it exists only to keep track of¬†an allocated chunk of memory. Think of it as TMemoryStream without any read or write functionality.

But how do we access that memory? Well this is where things get interesting because JavaScript has a view object that you attach to your buffer. Its a bit like creating a TStreamWriter or TStreamReader object under object pascal.

The interesting part is not that a view object expose read and write methods, but that the view type defines a fixed datatype. A fixed datatype of your choosing.

Lets say you want to access your buffer as an array of longwords, well there is a view for that (and you can also create something called “typed arrays” connected to a buffer). But you can also access the same buffer using say a word (16 bit) integer if you wish. Since you can have multiple views attached to the same buffer, this opens up for some interesting optimization techniques – not to mention spectacular error messages when you get it wrong (ps: stick to the RTL and it will take care of you).

The rule is that unsigned datatypes (like uint8 for a “normal” byte) is prefixed with a “u”, while signed types have no prefix.

  • So views that access a buffer as unsigned types are prefixed with “u”
    • uint8buffer
    • uint16buffer [and so on]
  • Signed views lack the prefix
    • int8buffer
    • int16buffer

Now the Smart Pascal RTL will shield you from all the nitty-gritty – but it’s important to understand how memory and raw data is dealt with behind the scenes by the JSVM. And abstracting the reader writer methods from the actual data makes sense. In fact I implemented something similar myself earlier (my ByteRage Delphi library).

Once you get to know it, every piece falls into place.

Smart memory

The units we will be looking at are the following:

  • System.Types
  • System.Types.Convert
  • System.Memory
  • System.Memory.Allocation
  • System.Memory.Buffer

Start by opening up the System.Memory unit. Spend a few minutes getting an overview of the classes and functions there. And when you have had a peek focus on the class TAddress.

TAddress¬†is a class that wraps around a JavaScript untyped buffer. It can be compared to a pointer holding the address to the memory you allocated (but also more, like the size). Let’s have a look at it:

  TAddress  = partial class(TObject)
  private
    FOffset:    Integer;
    FBuffer:    TMemoryHandle;
  protected
    function    GetSize: integer;virtual;
  public
    Property    Entrypoint: integer read FOffset;
    Property    Segment: TMemoryHandle read FBuffer;
    Property    Size: integer read GetSize;
    function    Addr(const Index: integer): TAddress;
    Constructor Create(const Segment: TMemoryHandle;
                const Offset: integer); overload; virtual;
    Destructor  Destroy;Override;
  end;

As you can see its very small and compact. It exposes a segment handle property which is a reference to the memory object it represents. It also expose size information, entrypoint and a curious Addr() function.

The Addr() function is for getting a sub address from within the memory buffer. Let’s say you want to write some data at position 1024 in your allocated buffer, well by calling FMyAddress.Addr(1023) it would return a new TAddress instance pointing at that particular place inside the buffer.

This is why we have an entrypoint property. This keeps track of what starting offset a TAddresse should start at. If you start at 1023, then all subsequent offsets will have the entrypoint added to them. They will be relative to this position. Think of it as the position property in TStream if you like. And you can naturally have as many TAddress instances pointing to the same buffer as you like – hence we use offsets to speed things up (instead of creating a new view object for every single reference).

Let’s allocate some memory:

procedure TForm1.ReserveMemory;
var
  LData: TAddress;
begin
  LData := TMarshal.Allocmem(1024 * 1024);
end;

The above code would allocate 1 megabyte of data, and LData now points to it. We can now use the other methods of TMarshal to work with the buffer. Let’s have a look at the methods TMarshal has to offer:

  TMarshal = class static
  public
    class property  UnManaged: TUnManaged;
    class function  AllocMem(const Size:Integer): TAddress;
    class procedure FreeMem(Const Segment: TAddress);

    class procedure Move(const Source:TAddress;
                    const Target:TAddress;
                    const Size:Integer);overload;

    class procedure Move(const Source:TMemoryHandle;
                    const SourceStart:Integer;
                    const Target:TMemoryHandle;
                    const TargetStart:Integer;
                    const Size:Integer);overload;

    class procedure FillChar(const Target:TAddress;
                    const Size:Integer;
                    const Value:char);overload;

    class procedure FillChar(const Target:TAddress;
                    const Size:Integer;
                    const Value:Byte);overload;

    class procedure ReAllocMem(var Segment: TAddress;
                    const Size:Integer);

    class function  ReadMemory(const Segment: TAddress;
                    const Size:Integer):TByteArray;overload;

    class procedure WriteMemory(const Segment: TAddress;
                    const Data:TByteArray);

    class procedure Fill(Const Buffer: TMemoryHandle; Offset: Integer;
                    ByteLen: Integer;const Value: Byte);
  end;

As you can see you pretty much have the same features as Delphi or Freepascal, albeit in a different form. Delphi being a native language doesnt need such intermediate objects, but the above code is actually leaps and bounds easier to work with than anything JavaScript gives you out of the box.

Lets say I want to write a longword 1024 bytes into our allocated memory. You would then use the Addr() function we talked about earlier to simplify things:

procedure TForm1.ReserveMemory;
var
  LData: TAddress;
begin
  LData := TMarshal.Allocmem(1024 * 1024);
  TMarshal.WriteMemory(LData.Addr(1023), TDataType.Int32ToBytes($BABECAFE));
end;

Now you are probably wondering – why waste time on converting the $BABECAFE value to bytes? Well you dont have to, but it’s one way of doing it. You can ofcourse create a uint32 view and write it directly like you would an array. But in order to make you more familiar with the RTL landscape I figured it would be valuable to start with the byte conversion. Which by the way saves you a lot of time. So have a peek at TDataType in System.Types.Convert.pas, the features there will impress you once you realize how little JavaScript gives you- and how much the RTL makes of it.

Moving up to TBinaryData

Now messing around with low-level byte buffers can be rewarding, but its not really the level you want to work on. You want the power of low-level stuff but with the infrastructure of high-level programming.

In the unit “system.memory.buffer” you will find a class that is written to cover all of your binary needs. It is optimized for reading and writing data fast and also writing various datatypes directly into the buffer without TAddress being a factor.
So while you can enjoy the low-level functionality of allocmem – you are actually expected to use TBinaryData. TMemoryStream is also fine, but TBinaryData will always be faster.

And it comes with methods for pretty much everything you could think of. Including portabilty! Be it buffer to stream, buffer to arrays or the other way around. TBinaryData has a lot of cool features. With ZLib just added to the new RTL – compression and zip file stuff will be added to this class (through partial classes, so only when you include the zlib unit will the methods appear in the class):

  TBinaryData = partial class(TAllocation
      ,IBinaryDataReadAccess
      ,IBinaryDataWriteAccess
      ,IBinaryDataReadWriteAccess
      ,IBinaryDataBitAccess
      ,IBinaryDataImport
      ,IBinaryDataExport)
  private
    FDataView:  JDataView;
  protected
    function    GetByte(const Index: integer):Byte;
    procedure   SetByte(const Index: integer;const Value:Byte);
  protected
    function    GetBitCount: integer; virtual;
    function    GetBit(const bitIndex: integer): boolean;
    procedure   SetBit(const bitIndex: integer;const value: boolean);
  protected
    function    OffsetInRange(Offset: integer): boolean;
  protected
    procedure   HandleAllocated; override;
    procedure   HandleReleased; override;
  public
    property    BitCount:integer read GetBitCount;
    property    Bytes[const ByteIndex: integer]: byte read GetByte write SetByte; default;
    property    Bits[const BitIndex: integer]: boolean read GetBit write SetBit;

    function    Allocation: IAllocation;

    procedure   AppendBytes(const Bytes: TByteArray);virtual;
    procedure   AppendStr(const Text: string);virtual;
    procedure   AppendMemory(const Buffer: TBinaryData; const ReleaseBufferOnExit: boolean);virtual;
    procedure   AppendBuffer(const Raw: TMemoryHandle);overload;
    procedure   AppendFloat32(const Value: Float32);virtual;
    procedure   AppendFloat64(const Value: Float64);virtual;

    procedure   CopyFrom(const Buffer: TBinaryData; const Offset: integer; const ByteLen: integer);
    procedure   CopyFromMemory(const Raw: TMemoryHandle;
                Offset: integer; ByteLen: integer);

    function    CutBinaryData(Offset: integer;ByteLen: integer): TBinaryData;
    function    CutStream(const Offset: integer; const ByteLen: integer): TStream;
    function    CutTypedArray(Offset: integer; ByteLen: integer): TMemoryHandle;

    procedure   Write(const Offset: integer; const Data: TByteArray);Overload;
    procedure   WriteFloat32(const Offset: integer; const Data: Float32);
    procedure   WriteFloat64(const Offset: integer; const Data: Float64);

    procedure   Write(const Offset:integer;const Data:TBinaryData);overload;
    procedure   Write(const Offset:integer;const Data:TMemoryHandle);overload;
    procedure   Write(const Offset:integer;const Data:String);overload;
    procedure   Write(const Offset:integer;const Data:integer);overload;
    procedure   Write(const Offset:integer;const Data:Boolean);overload;

    function    ReadFloat32(Offset:integer):Float;virtual;
    function    ReadFloat64(Offset:integer):Float;virtual;
    function    ReadBool(Offset:integer):Boolean;Overload;
    function    ReadInt(Offset:integer):integer;overload;
    function    ReadStr(Offset:integer;ByteLen:integer):String;overload;
    function    ReadBytes(Offset:integer;ByteLen:integer):TByteArray;overload;

    function    Clone: TBinaryData;

    procedure   SaveToStream(const Stream: TStream);
    procedure   LoadFromStream(const Stream: TStream);

    procedure   FromBase64(FileData: string);
    function    ToBase64: string;
    function    ToString: string;
    function    ToTypedArray: TMemoryHandle;
    function    ToBytes: TByteArray;
    function    ToStream: TStream;
    function    ToHexDump(BytesPerRow: integer;
                Options: TBufferHexDumpOptions): string;

    {$IFDEF APPTYPE_VISUAL}
    procedure   LoadFromFile(const aFileURI:String; const OnReady:TNotifyEvent);
    {$ENDIF}

    Constructor Create(aHandle: TMemoryHandle); overload; virtual;
  end;

If you already have allocated memory, or perhaps your server has been given a handle to a buffer for a file – notice that the constructor is overloaded and can work with existing buffer objects. In such a case it will not release the memory even if you free the instance (since it doesnt own it, only borrows it).

I took it one step further and added bit-manipulation as well. So you can allocate a large buffer and process it on¬†bit level if you need to. I actually used that a lot when doing some audio code. It is also very handy when keeping track of visible sprites in a game (“find me an idle sprite” where each bit represents a sprite’s busy state). And its even an important feature when writing database engines.

Well this was a quick introduction to low-level memory stuff in Smart. There is a ton of features I havent even mentioned yet, like being able to turn data into a URI object – that can be attached to anchor (A tag) objects and auto clicked to force a “save as” to appear. This is really neat if you have a web application that generates data or binary files the user can save.

Until next time !