Json.Net not serializing decimals the same way twice

Question

I was testing Json.NET serialization of a shopping cart I'm working on and noticed that when I serialize -> deserialize -> serialize again, I'm getting a difference in the trailing zero formatting of some of the decimal fields. Here is the serialization code:

private static void TestRoundTripCartSerialization(Cart cart)
{
    string cartJson = JsonConvert.SerializeObject(cart, Formatting.Indented);

    Console.WriteLine(cartJson);

    Cart cartClone = JsonConvert.DeserializeObject<Cart>(cartJson);

    string cloneJson = JsonConvert.SerializeObject(cartClone, Formatting.Indented);

    Console.WriteLine(cloneJson);

    Console.WriteLine("\r\n Serialized carts are " + (cartJson == cloneJson ? "" : "not") + " identical");
}

The Cart implements IEnumerable<T> and has a JsonObjectAttribute to allow it to serialize as an object, including its properties as well as its inner list. The decimal properties of Cart do not change, but some of the decimal properties of objects and their inner objects in the inner list/array do as in this excerpt from output of the code above:

First time serializing:

      ...
      "Total": 27.0000,
      "PaymentPlan": {
        "TaxRate": 8.00000,
        "ManualDiscountApplied": 0.0,
        "AdditionalCashDiscountApplied": 0.0,
        "PreTaxDeposit": 25.0000,
        "PreTaxBalance": 0.0,
        "DepositTax": 2.00,
        "BalanceTax": 0.0,
        "SNPFee": 25.0000,
        "cartItemPaymentPlanTypeID": "SNP",
        "unitPreTaxTotal": 25.0000,
        "unitTax": 2.00
      }
    }
  ],
 }

Second time serializing:

      ...
      "Total": 27.0,
      "PaymentPlan": {
        "TaxRate": 8.0,
        "ManualDiscountApplied": 0.0,
        "AdditionalCashDiscountApplied": 0.0,
        "PreTaxDeposit": 25.0,
        "PreTaxBalance": 0.0,
        "DepositTax": 2.0,
        "BalanceTax": 0.0,
        "SNPFee": 25.0,
        "cartItemPaymentPlanTypeID": "SNP",
        "unitPreTaxTotal": 25.0,
        "unitTax": 2.0
      }
    }
  ],
 }

Notice the Total, TaxRate, and some of the others have changed from four trailing zeroes to a single trailing zero. I did find some stuff regarding changes to handling of trailing zeroes in the source code at one point, but nothing that I understood well enough to put together with this. I can't share the full Cart implementation here, but I built a bare bones model of it and couldn't reproduce the results. The most obvious differences were my bare bones version lost some additional inheritance/implementation of abstract base classes and interfaces and some generic type usage on those (where the generic type param defines the type of some of the nested child objects).

So I'm hoping without that someone can still answer: Any idea why the trailing zeroes change? The objects appear to be identical to the original after deserializing either JSON string, but I want to be sure there isn't something in Json.NET that causes a loss of precision or rounding that may gradually change one of these decimals after many serialization round trips.

---

Updated

Here's a reproducible example. I thought I had ruled out the JsonConverter but was mistaken. Because my inner _items list is typed on an interface, I have to tell Json.NET which concrete type to deserialize back to. I didn't want the actual Type names in the JSON so rather than using TypeNameHandling.Auto, I've given the items a unique string identifier property. The JsonConverter uses that to choose a concrete type to create, but I guess the JObject has already parsed my decimals to doubles? This is maybe my 2nd time implementing a JsonConverter and I don't have a complete understanding of how they work because finding documentation has been difficult. So I may have ReadJson all wrong.

[JsonObject]
public class Test : IEnumerable<IItem>
{
    [JsonProperty(ItemConverterType = typeof(TestItemJsonConverter))]
    protected List<IItem> _items;

    public Test() { }

    [JsonConstructor]
    public Test(IEnumerable<IItem> o)
    {
        _items = o == null ? new List<IItem>() : new List<IItem>(o);
    }

    public decimal Total { get; set; }

    IEnumerator IEnumerable.GetEnumerator()
    {
        return _items.GetEnumerator();
    }

    IEnumerator<IItem> IEnumerable<IItem>.GetEnumerator()
    {
        return _items.GetEnumerator();
    }
}

public interface IItem
{
    string ItemName { get; }
}

public class Item1 : IItem
{
    public Item1() { }
    public Item1(decimal fee) { Fee = fee; }

    public string ItemName { get { return "Item1"; } }

    public virtual decimal Fee { get; set; }
}

public class TestItemJsonConverter : JsonConverter
{
    public override bool CanConvert(Type objectType) { return (objectType == typeof(IItem)); }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        object result = null;

        JObject jObj = JObject.Load(reader);

        string itemTypeID = jObj["ItemName"].Value<string>();

        //NOTE: My real implementation doesn't have hard coded strings or types here.
        //See the code block below for actual implementation.
        if (itemTypeID == "Item1")
            result = jObj.ToObject(typeof(Item1), serializer);

        return result;
    }

    public override bool CanWrite { get { return false; } }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer) { throw new NotImplementedException(); }
}

class Program
{
    static void Main(string[] args)
    {
        Test test1 = new Test(new List<Item1> { new Item1(9.00m), new Item1(24.0000m) })
        {
            Total = 33.0000m
        };

        string json = JsonConvert.SerializeObject(test1, Formatting.Indented);
        Console.WriteLine(json);
        Console.WriteLine();

        Test test1Clone = JsonConvert.DeserializeObject<Test>(json);
        string json2 = JsonConvert.SerializeObject(test1Clone, Formatting.Indented);
        Console.WriteLine(json2);

        Console.ReadLine();
    }
}

Snippet from my actual converter:

if (CartItemTypes.TypeMaps.ContainsKey(itemTypeID))
    result = jObj.ToObject(CartItemTypes.TypeMaps[itemTypeID], serializer);

Answer 1

If your polymorphic models contain decimal properties, in order not to lose precision, you must temporarily set JsonReader.FloatParseHandling to be FloatParseHandling.Decimal when pre-loading your JSON into a JToken hierarchy, like so:

public class TestItemJsonConverter : JsonConverter
{
    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        object result = null;

        var old = reader.FloatParseHandling;
        try
        {
            reader.FloatParseHandling = FloatParseHandling.Decimal;

            JObject jObj = JObject.Load(reader);
            string itemTypeID = jObj["ItemName"].Value<string>();

            //NOTE: My real implementation doesn't have hard coded strings or types here.
            //See the code block below for actual implementation.
            if (itemTypeID == "Item1")
                result = jObj.ToObject(typeof(Item1), serializer);
        }
        finally
        {
            reader.FloatParseHandling = old;
        }

        return result;
    }

Demo fiddle here.

Why is this necessary? As it turns out, you have encountered an unfortunate design decision in Json.NET. When JsonTextReader encounters a floating-point value, it parses it to either decimal or double as defined by the above-mentioned FloatParseHandling setting. Once the choice is made, the JSON value is parsed into the target type and stored in JsonReader.Value, and the underlying character sequence is discarded. Thus, if a poor choice of floating-point type is made, it's difficult to correct the mistake later on.

So, ideally we would like to choose as a default floating-point type the "most general" floating point type, one that can be converted to all others without loss of information. Unfortunately, in .Net no such type exists. The possibilities are summarized in Characteristics of the floating-point types:

As you can see, double supports a larger range while decimal supports a larger precision. As such, to minimize data loss, sometimes decimal would need to be chosen, and sometimes double. And, again unfortunately, no such logic is built into JsonReader; there is no FloatParseHandling.Auto option to choose the most appropriate representation.

In the absence of such an option or the ability to load the original floating-point value as a string and re-parse it later, you will need to hardcode your converter with an appropriate FloatParseHandling setting based upon your data model(s) when you pre-load your JToken hierarchy.

In cases where your data models contain both double and decimal members, pre-loading using FloatParseHandling.Decimal will likely meet your needs, because Json.NET will throw a JsonReaderException when attempting to deserialize a too-large value into a decimal (demo fiddle here) but will silently round the value off when attempting to deserialize a too-precise value into a double. Practically speaking, it's unlikely you will have floating-point values larger than 10^28 with more than 15 digits of precision + trailing zeros in the same polymorphic data model. In the unlikely chance you do, by using FloatParseHandling.Decimal you'll get an explicit exception explaining the problem.

Notes:

• I don't know why double was chosen instead of decimal as the "default default" floating point format. Json.NET was originally released in 2006; my recollection is that decimal wasn't widely used back then, so maybe this is a legacy choice that was never revisited?

• When deserializing directly to a decimal or double member, the serializer will override the default floating-point type by calling ReadAsDouble() or ReadAsDecimal(), so precision is not lost when deserializing directly from a JSON string. The problem only arises when pre-loading into a JToken hierarchy then subsequently deserializing.

• Utf8JsonReader and JsonElement from system.text.json, Microsoft's replacement for Json.NET in .NET Core 3.0, avoid this problem by always maintaining the underlying byte sequence of a floating-point JSON value, which is one example of the new API being an improvement on the old.

  If you actually have values larger than 10^28 with more than 15 digits of precision + trailing zeros in the same polymorphic data model, switching to this new serializer might be a valid option.