Archive for December 15, 2016

JSON Persistence, understanding it

December 15, 2016 Leave a comment

Two days ago I posted what I believe to be an elegant solution to general, JSON based, object persistency. It was naturally never meant to a “one storage mechanism to rule them all” type post; nor solution for that matter. But it did present the solution to a problem I know many people have been struggling with or had issues with quite commonly. It’s one of those issues that you never really see until you get there, until you sit down to code something that happens to include JSon serialization. And it bugs you because things like this should really be a one liner.

To clarify some of the responses I have recieved, both here and on Facebook, I figured it would be appropriate to do a follow-up.  I was quite tired when I posted that code, and I also realize that if you havent really bothered with JSON storage before (or indeed, looked at alternatives to Delphi’s somewhat old-fashioned, binary only built-in mechanism) then you might miss my point completely.

For instance, one person posted “what do you do with methods or properties that are not defined, just throw them away”. At first I frankly did not understand what the person was talking about – but then I realized that he had confused my talk about compilers and parsers and the fact that the code came from a library built for that -for the very purpose and point of the article.

Ofcourse I don’t throw away methods or properties, but that has nothing to do with the actual article. The article was really about finding a simple, elegant way of storing an object instance as JSON. Preferably using nothing but what Delphi has to offer.

What is the problem?

Before we look at the problem directly, let’s get an overview of the situation. For instance, how do you store a Delphi object instance as JSon? What are the options we have to work with? What exactly does Delphi provide out of the box?

These are fair, simple questions. Yet if you google it or look it up on Stack Overflow, the answers are overwhelmingly diverse. If you take into account libraries like SuperObject which people seem to mix and match with built-in classes from the runtime library, finding a simple reply can be a challenge. Especially if you don’t know that much about JSon to begin with or are learning Delphi from scratch.

Now to cut a long story short, Delphi has some outstanding solutions with regards to JSon. But naturally there are a few aspects here and there that could be better. And that is not really criticism, an RTL is a complex thing to write and maintain. And the level of support Delphi has for almost everything imaginable is staggering. So I don’t feel this is so much a “lack” as it is an oversight or prioritization issue.

What is lacking is the bridge, or gap if you will, between the world of natively compiled assembly code objects in memory (read: ordinary object pascal objects), and easily representing these as a text-based, Json (JavaScript object notation) format.

To my knowledge there are only two methods you can use to “save” or “take a snapshot” of an object instance that results in Json. The first being:

LSomeObjVar := TJSon.ObjectToJsonObject(instance);

This essentially enumerates all the published properties of that instance and stores it directly in a JSon object (which is itself an instance) and returns that. Which is very handy if you are building a larger JSon schema containing many objects.

The second method, which does more or less the same is this:

LSomeStrVar := TJSon.ObjectToJsonString(instance);

As you no doubt guess that returns the actual JSon text rather than a JSon object. Which is even more handy since that’s ultimately what we want.

To be fair, extracting JSon information through the use of the VCL or FMX is not really a problem. It does what is expected in traditional manner, namely to enumerate published properties that are both readable and writable and emit these in the format in question.

Anyone familiar with Delphi’s Assign(), AssignTo() and the DefineProperty() methods, the latter being a topic of great confusion to beginners; and finally TFiler which is used to deal with data that cannot easily be represented by ordinary properties. If you have any knowledge of these then you should have intimate knowledge of how it all works.

Right, that was the “stringify” options at hand. Now let’s look at the second part of the magic trick – namely to reincarnate an instance from JSon text.

Parsing JSon into an instance

I may be missing something but to my knowledge there are only two methods we can use for this. Naturally you can choose to traverse an object yourself through RTTI, save the information as JSon, then assemble it by hand at the other end — or some variation of the two; but the keyword here were simple, elegant and at-hand. We all know that anything is possible if you sit down and allocate enough time and energy – but this was about what we can do out of the box.

The first method takes a JSon object containing the data, but it demands a class in order to know what instance to create:


This is, in essence, the problem. Knowing beforehand the classtype to use.

It is a problem if you have several classes that inherit from a common ancestor. For example, let’s say you are making an invoice program. And let’s say you have a base-class simply called TInvoice. Inheriting from that is TRetractedInvoice and TPaidInvoice.

  • TInvoice
    • TRetractedInvoice
    • TPaidInvoice

TRetractedInvoice being a class that represents an invoice where where the customer has gotten his money back. Perhaps the product was damaged, did not match his expectations or something similar. Either way, this class is used to represent invoiced where the money has been returned. This is noted in the ledger for the books to even out. Otherwise there will be hell to pay should you get an audit.

The second class that inherits from TInvoice is TPaidInvoice. This decendant represents an invoice which has been paid in full. The transaction is complete and there is nothing to report. Just a normal sale.

Now you could argue that using classes to represent states is overkill. You could in fact just have TInvoice and then use a datatype to represent these states. But that’s not the point. You should be allowed to use classes like this, it should not even be a debate.

But how exactly do you know what class to use before the actual data has been read? No matter how you look at this problem, you cannot escape the fact that seeing into the future is reserved for circus sideshows and scam artists abusing peope’s insecurities.

Thankfully, TJSon presents a second alternative, namely one where you give it a pre-created instance and it will map the properties that match. So if a property in the JSon data match a property in the instance, it will populate it.

TJSon.JsonToObject(someinstance, Source);

But wait, that still does not solve our problem! Simply providing an a pre-made instance gives us more options ofcourse, but the problem of foresight and being able to look into the future is still there.

Unless, we make sure that looking into the past preserves the future that is.

The solution

The obvious solution is to simply store the classname with the JSON text when we stringify the instance. And that is precisely what I have done. This means ofcourse that we have to read the JSon data before we call JSonToObject() but honestly, that is a small price to pay for complete uniformity.

But this brings up a couple of other challenges, namely:

  • How do we create a class based purely on a string-name?
  • How do we avoid writing the same information for each step in the inheritance chain

When it comes to the first topic, this is where Registerclass() comes into play.

Every class you register through Registerclass() can also be created by name and it’s type looked up. This is an essential part of the binary, built-in persistant system Delphi has always shipped with. So we can put some faith in it.

Interestingly, when looking at these topcis, I suddenly realized something important. It took me a second to see that I was in fact imposing a condition that did not exist, not unless you wanted to clone instances. Because who will be doing the parsing and reading anyhow? There will already be an instance created to this exact piece of data: namely none other than “self”. This is why my solution contains:

procedure TQTXASTPersistent.Parse(ObjectData: string);
  LSchema: TJsonObject;
  LObjData: TJsonObject;
  LEntry: TJSonObject;
  LId: string;
  LId := '';

  // Parse whole schema
  LSchema := TJSonObject( TJSonObject.ParseJSONValue(ObjectData, true) );
    if LSchema.Values[QualifiedClassName] <> nil then
      // Find storage entry for our class
      LEntry := TJsonObject( LSchema.GetValue(QualifiedClassName) );

      // attempt to get the identifier
      if LEntry.Values['$identifier'] <> nil then
      LId := LEntry.GetValue('$identifier').Value;

      // validate identifier
      if LId.Equals(Classname) then

        // Grab the data chunk of our entry
        LObjData := TJSonObject( LEntry.GetValue('$data') );

        // Data PTR valid?
        if LObjData <> nil then

          // Map values into our instance
            on e: exception do
            raise EQTXASTJSONError.CreateFmt
            ('Serialization failed, system threw exception %s with message "%s" error',
            [e.ClassName, e.Message]);

        end else
        raise EQTXASTJSONError.CreateFmt
        ('Serialization failed, unable to find section ["%s\$data"] in JSON document error',

      end else
      raise EQTXASTJSONError.CreateFmt
      ('Serialization failed, invalid signature, expected %s not %s error',
      [classname, LId]);

    end else
    raise EQTXASTJSONError.CreateFmt
    ('Serialization failed, unable to find section ["%s"] in JSON document error',

procedure TQTXASTPersistent.ReadObjectJSon(const Source: TJSONObject);
  TJSon.JsonToObject(self, Source);

Simple, elegant and to the point.

It is also flexible to avoid the second topic mention above, namely the trap of recursive or repeatative data. So you can inherit from this class all you want, it will never produce JSon for it’s ancestor. It will always produce the full set of published properties of it’s datatype. Exclusively. So unless you override and mess about with how data is written, it will work quite elegantly for ordinary tasks.

As a safety meassure, I store the object information in its own namespace. I use the qualified name of the class as an entrypoint, and the data for that classtype is stored there. If you should inherit from the baseclass and introduce some special data (for example by overriding the WriteObjectJSon() and ReadObjectJSon() methods, that will automatically end up in it’s own namespace or entrypoint. Pretty cool if I say so myself 🙂

Let’s look at a simple example. Here I have a very humble class with two published properties. I changed the name TQTXASTPersistent to just TQTXPersistent. Just to make it simpler to read:

TTestClass = class(TQTXPersistent)
  FFirst: string;
  FLast:  string;
  property FirstName: string read FFirst write FFirst;
  property LastName: string read FLast write FLast;

When we create an instance of this, put something in the properties, then serialize. This is how the serialized data looks like:


If I for some reason need to inherit from that again, and alter the storage routine manually to include more data, The “mainform.TTestClass” section will still be there, but now the properties particular to my new class will end up in its own section (!). They will never collide or overwrite each other. And the root property $classname$ will always contain the actual name of the class the data represents, never an ancestor. This is because inherited data is written first, and the actual instance will write it’s information last.

So here we have a pretty good system if you ask me. A lot easier to work with and deal with than manually having to traverse RTTI information. And even better, no external libraries required. Just what Delphi gives us out of the box.

Cloning, how did you solve it?

But the problem still remains. Namely the abillity to create an instance purely based on the information in the JSon data. Providing the class is known to delphi (or that I write some registerclass equivalent, but why recreate the wheel?).

Well, turned out that with the above mechanism in place, it was quite easy. Instead of a simple “clone” method I made it even more generic, a class function, allowing you to create any instance from a JSon file – providing the class has been registered with Delphi first:

class function TQTXASTPersistent.JSONToObject
      (const MetaData: string; var obj: TObject): boolean;
  LSchema: TJsonObject;
  LClassName: string;
  LType: TClass;
  LNameNode: TJSONValue;
  LObj: TQTXASTPersistent;
  result := false;
  obj := nil;

  // Parse whole schema
  LSchema := TJSonObject( TJSonObject.ParseJSONValue(MetaData, true) );
    LNameNode := LSchema.Values['$classname$'];
    if LNameNode <> nil then
      LClassName := LNameNode.Value;
      LType := GetClass(LClassName);
      if LType <> nil then
        LObj := TQTXASTPersistent( LType.Create );
        obj := LObj;
        result := true;

Well, I hope this clears up any confusion on the subject. In my own defence I was very tired when I posted the code, which is always a bad idea.

And as always, if you know of a better way, perhaps some command i have missed or a class I have overlooked, I would love to learn about it.