Dreamstate Coding: 2018

Saturday, December 8, 2018

F# typed tuples and how to pattern match them

Noticed a warning fly-by in the compiler log in Visual Studio

warning FS0026: This rule will never be matched

When I opened the location that it complained about I saw the following code from my main event reducer:

let HandleEvent (state:State) (event:ApplicationEvent) : State =
 match event with 
 | ApplicationEvent (h, p) -> 
  {
   CurrentPlayer = CurrentPlayerReducer.HandleEvent state h p
   LastEventId = Some h.Id
   Timestamp = h.Timestamp
   PlayerStore = PlayerReducer.HandleEvent state.PlayerStore h p
   SpaceBodyStore = SpaceBodyReducer.HandleEvent state.SpaceBodyStore h p
  }
 | _ -> state

The warning was that the _ -> state rule will never match and I started to wonder if there might be a better way to write this thing.
The ApplicationEvent is defined as follows:

type ApplicationEvent = ApplicationEvent of EventHeader * obj

For those of you that have read my previous posts about Event Sourcing in F# may notice that the event definition has evolved a little bit since my last post in that series, no worries, I will post more when I have something working.

As you can see the ApplicationEvent type is pretty much just a discriminated union of 1 that has a named tuple.
This all makes it easy to handle the application event in the system as one object but when you want to look at the header and payload separately you will have to pattern match it to extract its parts, hence the pattern matching in the function above.

After a little bit of thinking I tried the following instead:

let HandleEvent (state:State) (ApplicationEvent (h, p)) : State =
 {
  CurrentPlayer = CurrentPlayerReducer.HandleEvent state h p
  LastEventId = Some h.Id
  Timestamp = h.Timestamp
  PlayerStore = PlayerReducer.HandleEvent state.PlayerStore h p
  SpaceBodyStore = SpaceBodyReducer.HandleEvent state.SpaceBodyStore h p
 }

I.e. I pattern match already in the in parameters to the function and thus get rid of the whole match construct in the function, and it works!
A day when you learn something new is not a wasted day!

So hope this helps someone out there!

Wednesday, October 17, 2018

Teaching myself Unity - Logging to screen

I've played around quite a lot with OpenGL and built my own engine to some extent (as you can see in other posts on this blog). But I want to create games, doing that with a home-written engine puts a lot of focus on the engine and very little on the actual game. So I decided to try out Unity as it seems popular and has nice integration with Visual Studio

Getting started

Install both and start Unity.

First screen in Unity. Just click on the New button to the upper right to create our first project

Name the project and select where it should be placed on disk. I also disabled unity analytics as I don't think that this test project would have any usage for it.

Configure script editor

Next step is to make sure that Unity thinks that Visual Studio should be used for script editing

Go to Edit menu -> Preferences and in the popup select External Tools and make sure that External Script Editor is set to Visual Studio 2017 (Community). Or any other version of Visual Studio if you have the licensed versions

Logging to screen

I like to have stuff happening on the screen, and in my mind the first step should be to add logging to screen so that we can follow what happens in the game without having to tab out to the Unity editor (or when we run the game outside of the development environment)

So, how to accomplish logging to screen?

In the project tab. Assets folder, create a new folder and name it Scripts

Create a new folder called scripts (see above picture)
Create a new C# script in that folder and call it Logger

In the SampleScene create a new empty GameObject and call it Screen Logger

Create a new GameObject and call it Screen Logger
Select the Screen Logger GameObject and drag the Logger script to the Inspector view for the screen logger to attach the script to the GameObject

Screen Logger Game Object with attached Logger script in the Inspector view

Next, let's create the UI component that we will log to.

Right click in the Sample Scene tab and select UI -> Text
Rename the newly created Text to Log

Scene view with the Log Text control (New Text), 2D view selected in the upper left corner and the white line is the borders of the canvas

Go to the Scene view and click on the 2D button at the top so that we can position our UI component
Select the Log Game Object and move your mouse cursor over the Scene view and press F on the keyboard. This will find and focus the view on that particular component.
Zoom out with your mouse-wheel so that you see the canvas borders

Resized textbox to fill the upper area of the canvas

Resize the textbox so that it fills the upper area of the canvas. Notice the color change of the canvas border when the textbox is at the border.

In the Inspector, set stretch and anchor to the top of the canvas

Select the text box and go to the inspector. Select stretch and anchor point according to the picture above

Logger script

So, now we have our UI component setup and placed at the top of the canvas area, lets look at the Logger script.

using System;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;

public class Logger : MonoBehaviour
{
    public static Logger Instance { get; private set; }
    public Text Log;
    private List<string> _logLines = new List<string>();

    void Start()
    {
        Instance = this;
    }

    void Update()
    {
        // test code to check that this works
        Write(Time.fixedTime.ToString());
    }

    public void Write(string text)
    {
        _logLines.Add(text);
        if (_logLines.Count > 3)
            _logLines.RemoveAt(0);
        Log.text = string.Join(Environment.NewLine, _logLines.ToArray());
    }
}

Each script is generated with a Start and Update method.
The static Logger Instance will help us access this script from outside (Singleton pattern). We assign it with this in the Start method (Start is guaranteed to only execute once)
Next we add a using UnityEngine.UI; line to the top and a public variable called Text Log. This will allow us access to an UI component in the scene. We will connect them in a later step, here we just say that we will be using a GameObject of type Text
After that we define a private variable that holds a list of strings that have been logger and initialize it inline
Now, create a new method called Write that takes a string as parameter.
Add the input parameter to the list of strings and then we make sure that the list doesn't grow by removing old items if there are more then 3 items in the collection.
Lastly we join all strings in the list with a new line between each and assign it to the text field of the Text UI component.

And that's pretty much it. You can of course play around with more information but this is the basic logging capability that I am after.

Notice the Update method calls the Write each frame, this is only for testing purposes and the whole Update method can be removed once we know that the script works.

Connect our Log text element to our Logger script in the Screen Logger game object inspection Click on the small circle to the right of the field to popup the Select Text dialog. It will show all GameObjects in the scene of the type Text.

So, our script is done. Now we just have to connect the correct Text component to it

In the inspector for Screen Logger, click on the small circle to the right of the Log field to popup the Select Text dialog
As we only have one Text component in the scene at the moment, just select it and close the popup

The end result of our logger in action, currently printing out the time each frame

So it works, you can now remove the Update method from the script as it is not really needed.

Usage from other scripts:

Logger.Instance.Write("information that we want to put on screen");

That's it!

2 great resources that helped me a lot were:

All code provided as-is. This is copied from my own code-base, May need some additional programming to work. Use for whatever you want, how you want! If you find this helpful, please leave a comment or share a link on social media, not required but appreciated! :)

Saturday, September 29, 2018

Immutability in .NET

First off, what are immutable objects

Immutable is pretty much a fancy word for unchangeable. In other words, once you create an object it will have the same properties for its entire life. No way to reassign for example a name, if you want to do that you would have to create a new object with the same properties but with the new name. Immutable = Not Mutable

Why bother?

I didn't understand the point of immutable objects, I though that as long as we encapsulate and follow good object oriented guidelines then we are good to go.. Right?
Let's look at a naive example of things that can go wrong:

public class CSharpPoco
{
    private uint _value;
    public uint Value => _value;
    public void Increment()
    {
        _value++;
    }
    public uint GetNext()
    {
        Increment();
        return _value;
    }
}
        
public class ServiceLayer
{
    private CSharpPoco _poco = new CSharpPoco();
    public CSharpPoco GetPoco()
    {
        return _poco;
    }
    public uint PreviewNextValue()
    {
        return _poco.GetNext();
    }
}

In the service layer we have 2 methods to call, get the poco and preview its next value.
The following service code would throw the side effect exception:

var service = new ServiceLayer();
var poco = service.GetPoco();
var firstValue = poco.Value;
var preview = service.GetPreviewValue();
if (preview <= firstValue)
    throw new Exception("preview can't be smaller then or equal to the previus value");
if (poco.Value == preview)
    throw new Exception("side effect");

Meaning, we get an unexpected side-effect by calling the second method in the service. It manages to change the value of the first service calls result. It was zero, but after the preview call it has been set to 1. This is quite a simple example, but after having debugged scenarios like these in production code with huge code bases.. You get the idea, a unexpected side effect is often a cause for hard-to-find bugs.
So, how could immutability have helped us here? once the var poco = service.GetPoco() was called, the result would never change. Preferably we would not be able to reasign the variable poco to other references either (similar to the const keyword in TypeScript or F# let). So instead of var I would have liked a const poco = service.

What options do we have in .NET?

Readonly objects in C# with constructor lists

public class CSharpImmutableConstructorList
{
    public readonly Guid Id;
    public readonly string Name;
    public readonly string DisplayName;
    public CSharpImmutableConstructorList(Guid id, string name, string displayName)
    {
        Id = id;
        Name = name;
        DisplayName = displayName;
    }

    public CSharpImmutableConstructorList SetDisplayName(string displayName)
    {
        return new CSharpImmutableConstructorList(Id, Name, displayName);
    }
}

Basically the magic here is done by the readonly keyword that states that only the constructor can set the value of the variable. Once it has been set, it can only be read. For larger objects, it can get a little heavy on the constructor and you have to supply all the values every time you want to change. For example the SetDisplayName function, it supplies the Id and Name even though they have the same value as before.
If we add more properties to this class, we would have to change all calls to the constructor and add the new properties for them as well. So a little heavy on the maintenance side.

Readonly objets in C# with Modify pattern

public class CSharpImmutableModifyPattern
{
    public readonly Guid Id;
    public readonly string Name;
    public readonly string DisplayName;
    public static CSharpImmutableModifyPattern Default { get; } = new CSharpImmutableModifyPattern(Guid.Empty, string.Empty, string.Empty);
    private CSharpImmutableModifyPattern(Guid id, string name, string displayName)
    {
        Id = id;
        Name = name;
        DisplayName = displayName;
    }
    public CSharpImmutableModifyPattern Modify(Guid? id = null, string name = null, string displayName = null)
    {
        return new CSharpImmutableModifyPattern
            (
                id ?? Id,
                name ?? Name,
                displayName ?? DisplayName
            );
    }
    public CSharpImmutableModifyPattern SetDisplayName(string displayName)
    {
        return Modify(displayName: displayName);
    }
}

This is pretty much the same pattern as I've described in a previous post (Functional adventures in .NET C# - Part 1, immutable objects), the difference is that we use the readonly members instead of get-only fields.
Also note that the constructor is set to private to prevent usage of it from outside, instead we will use the Default static get-only field that sets up a default invariant of the class and call the new method Modify on it.
The nice thing with the Modify pattern is that you don't have to supply anything else then the changed property as shown in the SetDisplayName method. Even if we add new parameters, we don't have to change the SetDisplayName method as it specifies that it only wants to supply the displayName and nothing else.

Record types in F#

module Techdump =
    open System
    type FSharpRecord =
        {
            Id : Guid
            Name : string
            DisplayName : string
        }
        member this.SetDisplayName displayName =
            { this with DisplayName = displayName }

The F# Record version of the same type as show before. Used from C# this type feels similar to the C# Constructor List version as you have to supply all the constructor parameters when calling new. The magic happens in the SetDisplayName function that takes a new displayName value and then calls the record constructor with the current record and whatever fields have changed. I.e. the with keyword in record construction. We get the power of CSharpImmutableModifyPattern, but without having to maintain a Modify method, it is all included in the language.

Performance

In this part we will look at the run-time performance the different immutable patterns described above.

Performance test source code

CSharpPoco

public class CSharpPoco
{
    public Guid Id { get; private set; }
    public string Name { get; private set; }
    public string DisplayName { get; private set; }
    public CSharpPoco(Guid id, string name, string displayName)
    {
        Id = id;
        Name = name;
        DisplayName = displayName;
    }

    public void SetDisplayName(string displayName)
    {
        DisplayName = displayName;
    }
}

Added a C# poco object so that we have a baseline to run against. I.e, here we just mutate the object. No new instance is created, this is the traditional way of changing a value in an object.

Performance tester

public class ImmutabilityPerformance
{
 public void Execute()
 {
  Log("--------------------------------------");
  Log("... ImmutabilityPerformance");
  ExecuteTest(1_000_000);
 }
 
 
 private void ExecuteTest(int iterations)
 {
  string s = string.Empty;
  try
  {
   var data = new List<string> { "Melissa Lewis", "Maya", "Smith", "Beverly Marsh", "Jane Vasko", "Molly Bloom" };
   var csharpPocoTimings = new RunningAverage();
   var fsharpRecordTimings = new RunningAverage();
   var csharpConstructorTimings = new RunningAverage();
   var csharpModifyTimings = new RunningAverage();
   for (int i = 0; i < iterations; i++)
   {
    var csharpPoco = new CSharpPoco(Guid.NewGuid(), "Jessica Chastain", "Jessica Chastain");
    var fsharpRecordOriginal = new Techdump.FSharpRecord(Guid.NewGuid(), "Jessica Chastain", "Jessica Chastain");
    var csharpConstructorOriginal = new CSharpImmutableConstructorList(Guid.NewGuid(), "Jessica Chastain", "Jessica Chastain");
    var csharpModifyOriginal = CSharpImmutableModifyPattern.Default.Modify(Guid.NewGuid(), "Jessica Chastain", "Jessica Chastain");

    for (int dataIndex = 0; dataIndex < data.Count; dataIndex++)
    {
     var item = data[dataIndex];


     csharpPocoTimings.Add(TimeAction(() =>
     {
      csharpPoco.SetDisplayName(item);
     }));
     if (csharpPoco != null)
      s = csharpPoco.DisplayName;

     Techdump.FSharpRecord fsharpRecordModified = null;
     fsharpRecordTimings.Add(TimeAction(() =>
     {
      fsharpRecordModified = fsharpRecordOriginal.SetDisplayName(item);
     }));
     if (fsharpRecordModified != null)
      s = fsharpRecordModified.DisplayName;

     CSharpImmutableConstructorList csharpConstructorModified = null;
     csharpConstructorTimings.Add(TimeAction(() =>
     {
      csharpConstructorModified = csharpConstructorOriginal.SetDisplayName(item);
     }));
     if (fsharpRecordModified != null)
      s = csharpConstructorModified.DisplayName;

     CSharpImmutableModifyPattern csharpModifyModified = null;
     csharpModifyTimings.Add(TimeAction(() =>
     {
      csharpModifyModified = csharpModifyOriginal.SetDisplayName(item);
     }));
     if (csharpModifyModified != null)
      s = csharpModifyModified.DisplayName;
    }
   }
   Log($"CSharpPoco\tIterations:\t{iterations}\tAverage:\t{csharpPocoTimings.Average:0.000}\tticks\tTotal:\t{csharpPocoTimings.Total:0.000}\tticks");
   Log($"FSharpRecord\tIterations:\t{iterations}\tAverage:\t{fsharpRecordTimings.Average:0.000}\tticks\tTotal:\t{fsharpRecordTimings.Total:0.000}\tticks");
   Log($"CSharpImmutableConstructorList\tIterations:\t{iterations}\tAverage:\t{csharpConstructorTimings.Average:0.000}\tticks\tTotal:\t{csharpConstructorTimings.Total:0.000}\tticks");
   Log($"CSharpImmutableModifyPattern\tIterations:\t{iterations}\tAverage:\t{csharpModifyTimings.Average:0.000}\tticks\tTotal:\t{csharpModifyTimings.Total:0.000}\tticks");
  }
  catch (Exception ex)
  {
   Log($"Fail\tDataCount\t2\tIterations:\t{iterations}\tFailed\t{ex.Message}");
  }
 }


 private float TimeAction(Action action)
 {
  var sw = Stopwatch.StartNew();
  action();
  sw.Stop();
  return sw.ElapsedTicks;
 }

 private void Log(string s)
 {
  Console.WriteLine(s);
  File.AppendAllText(@"c:\temp\enumToStringTest.txt", $"{s}{Environment.NewLine}");
 }
 
}

And the RunningAverage class from a previous post.

Results

ImmutabilityPerformance	Iterations	Average (ticks)	Total (ticks)
CSharpPoco	1000000	0.090	537402
FSharpRecord	1000000	0.121	727156
CSharpImmutableConstructorList	1000000	0.120	720220
CSharpImmutableModifyPattern	1000000	0.161	965545

Here we can see that the poco baseline test is the fastest by far. Using the F# Record type and C# locked down object with constructor are pretty much equal in execution. The modify pattern in the C# code really drops the performance but gives more readable code, at least if we have long constructor lists in objects.

Rewritten increment example

Let's go back and look at the first increment example again and try to rewrite it with immutability

public class CSharpImmutable
{
    public readonly uint Value;
    public CSharpImmutable(uint value)
    {
        Value = value;
    }
    public CSharpImmutable Increment()
    {
        return new CSharpImmutable(Value + 1);
    }
    public uint GetNext()
    {
        return Value + 1;
    }
}

And in F#

type FsharpIncrement =
    {
        Value : uint32    
    }
    member this.Increment () =
        { this with Value = this.Value + 1u }
    member this.GetNext () =
        this.Value + 1u

Conclusions

In my opinion Immutable objects are the way to go if you want to write maintainable code for the long run, what technique you choose is entirely up to you. There are probably other ways to achieve the same that I don't know about, if you have ideas please post a comment!

After I started writing F# code, and started getting all the immutability features for free and especially the with keyword for records has persuaded me to start writing all my core business models in F#.

let point' = { point with X = 2; }

In my eyes, that line of code just there, is so damn beautiful.

All code provided as-is. This is copied from my own code-base, May need some additional programming to work. Use for whatever you want, how you want! If you find this helpful, please leave a comment or share a link, not required but appreciated! :)

Friday, September 28, 2018

FSharp Enum ToString vs. C# Enum Dictionary Lookup for Localization

I was reading Tao Liu's post regarding enums with space for localization to save time and I just had to run a test if this really was faster then defining a lookup table?

On my computer

--------------------------------------
... FsharpEnumToString vs. CSharpEnumDictionaryLookup
FSharpToString  Iterations: 10000000;  Average: 0,287 ticks; Total: 14374450,000 ticks;
CSharpLookup    Iterations: 10000000;  Average: 0,134 ticks; Total: 6688315,000 ticks;

So, a simple dictionary lookup is still faster then ToString of a F# enum.
So I'll guess my team will keep the localization as data files / database table that is loaded at runtime into a variable. This way we can outsource the translations to other teams and specialists while keeping the code clean. A misspelled enum member by a tired developer does not result in a hotfix of an application with new builds etc, just a data file update in runtime.

The code used to perform this test:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using fsharp.techdump;

namespace techdump.console.Tests
{
    public class FsharpEnumToStringVsLookup
    {
        private Dictionary<CSharpEnum, string> _lookup = new Dictionary<CSharpEnum, string>
        {
            { CSharpEnum.Registration, "Registration" },
            { CSharpEnum.UnderReview, "Under Review" },
            { CSharpEnum.Approval, "Approval" },
            { CSharpEnum.Release, "Release" },
            { CSharpEnum.PostRelase, "Post Release" }
        };
        private enum CSharpEnum
        {
            Registration = 0,
            UnderReview = 1,
            Approval = 2,
            Release = 3,
            PostRelase = 4,
        }
        public void Execute()
        {
            Log("--------------------------------------");
            Log("... FsharpEnumToString vs. CSharpEnumDictionaryLookup");
            ExecuteTest(10_000_000);
        }
        
        
        private void ExecuteTest(int iterations)
        {
            string s = string.Empty;
            try
            {
                var index = new List<int> { 0, 1, 2, 3, 4 };
                var fsharpToStringTimings = new RunningAverage();
                var csharpLookupTimings = new RunningAverage();
                for (int i = 0; i < iterations; i++)
                {
                    for (int dataIndex = 0; dataIndex < index.Count; dataIndex++)
                    {
                        var item = index[dataIndex];
                        var fsharpEnumMember = (Techdump.FsharpEnum)item;
                        fsharpToStringTimings.Add(TimeAction(() =>
                        {
                            s = item.ToString();
                        }));
                        if (!string.IsNullOrEmpty(s))
                            s = string.Empty;
                        var csharpEnumMember = (CSharpEnum)item;
                        csharpLookupTimings.Add(TimeAction(() =>
                        {
                            s = _lookup[csharpEnumMember];
                        }));
                        if (!string.IsNullOrEmpty(s))
                            s = string.Empty;
                    }
                }
                Log($"FSharpToString\tIterations:\t{iterations}\tAverage:\t{fsharpToStringTimings.Average:0.000}\tticks\tTotal:\t{fsharpToStringTimings.Total:0.000}\tticks");
                Log($"CSharpLookup\tIterations:\t{iterations}\tAverage:\t{csharpLookupTimings.Average:0.000}\tticks\tTotal:\t{csharpLookupTimings.Total:0.000}\tticks");
            }
            catch (Exception ex)
            {
                Log($"Fail\tDataCount\t2\tIterations:\t{iterations}\tFailed\t{ex.Message}");
            }
        }


        private float TimeAction(Action action)
        {
            var sw = Stopwatch.StartNew();
            action();
            sw.Stop();
            return sw.ElapsedTicks;
        }

        private void Log(string s)
        {
            Console.WriteLine(s);
            File.AppendAllText(@"c:\temp\enumToStringTest.txt", $"{s}{Environment.NewLine}");
        }
        
    }
    
}

FSharp Enum defined as follows:

namespace fsharp.techdump

module Techdump =
    type public FsharpEnum =
        Registration = 0
        | ``Under Review``=1
        | Approval = 2
        | Release = 3
        | ``Post Release`` = 4

And the RunningAverage class from a previous post.

Wednesday, September 26, 2018

GeoCoordinateWatcher and GMap.NET

Playing around a little with the GMap.NET - Maps For Windows and figured I would use my computers location as a startup point for the map view.
Ended up encapsulating System.Device.Location.GeoCoordinateWatcher in a static class to get around the NaN coordinates returned from an uninitialized GeoCoordinateWatcher as it turns out that GMap.NET doesn't like NaN at all.

using System.Device.Location;

namespace dreamstatecoding.blogspot.com
{
    public static class GeoProvider
    {
        private static GeoCoordinateWatcher _watcher = new GeoCoordinateWatcher();
        public static bool IsInitialized { get; private set; }

        static GeoProvider()
        {
            _watcher.PositionChanged += _watcher_PositionChanged;
            _watcher.Start();
        }
        public static void Cleanup()
        {
            _watcher.Stop();
            _watcher.Dispose();
            _watcher = null;
        }
        private static void _watcher_PositionChanged(object sender, GeoPositionChangedEventArgs<GeoCoordinate> e)
        {
            IsInitialized = true; ;
        }

        public static GeoCoordinate GetCoordinates()
        {
            var coordinates = _watcher.Position.Location;
            if (double.IsNaN(coordinates.Latitude))
                return new GeoCoordinate(0, 0);
            return coordinates;
        }
    }
}

Usage: Giving the provider some time to get hold of the first coordinates. If it doesn't, the coordinate returned would be longitude:0;latitude:0. So pretty much in the ocean outside of Africa.

private void SetupMap()
{  
 for(int i = 0; i < 1000; i++)
 {
  if (GeoProvider.IsInitialized)
   break;
  Thread.Sleep(151);
 }
 var coordinates = GeoProvider.GetCoordinates();
 gmap.MapProvider = GMapProviders.GoogleMap;
 gmap.Position = new PointLatLng(coordinates.Latitude, coordinates.Longitude);
 gmap.MinZoom = 0;
 gmap.MaxZoom = 24;
 gmap.Zoom = 15;
}

It seems that the thread in the GeoCoordinateWatcher is not set to Background as default. So I just force kill it when the application shuts down for now. I put this in my Program.cs file in the WinForms project:

[STAThread]
static void Main()
{
 Application.ApplicationExit += Application_ApplicationExit;
 
 Application.EnableVisualStyles();
 Application.SetCompatibleTextRenderingDefault(false);
 Application.Run(new Form1());
}        
private static void Application_ApplicationExit(object sender, EventArgs e)
{
 GeoProvider.Cleanup();
}

The elegance of a F# WebRequest

Love the elegance of doing a WebRequest in F#

module WebLoader =
    open System.Net
    open System.IO

    let GetString (url:string) =
        let request = WebRequest.Create(url)
        use response = request.GetResponse()
        use stream = response.GetResponseStream()
        use reader = new StreamReader(stream)
        reader.ReadToEnd()

Especially the use keyword compared to the C# using { } blocks. Me like!

Wednesday, June 20, 2018

Functional Adventures in F# - Event sourcing lessons learned

Using domain objects in actions

The first actions that I used basically had me create the domain objects on the outside and then plug them into the create action. The store only took the object and placed it in the stores Map over objects.

type AddUser = { ActionId: Guid; User : User }
type UserStore =
 {
  Users: Map<UserId, User>
 }
 member this.AddUser (action:AddUser)=
  { this with Users = this.Users.Add(action.User.Id, action.User); }

 member this.HandleAction (action:obj) =
  match action with
  | :? AddUser as a -> this.AddUser a
  | _ -> this

This makes your domain model a part of the persistence layer (as all actions are persisted). I.e. you cannot change your model without getting nifty errors (like null exceptions)
You might think that: You still persist snapshots of the whole ApplicationState, i.e. the User object ends up in persistence.
True. But when you do 'model breaking changes', you could just remove all snapshots and then rerun all actions from the start and get the equivalent state as you did before the breaking change. If you in turn include the domain object in the actions, you are screwed.
A better way to write this would be:

type AddUser = { ActionId: Guid; UserId : UserId; Name:string; Email:string }
type UserStore =
 {
  Users: Map<UserId, User>
 }
 member this.AddUser (action:AddUser)=
  let user = 
   {
    Id = action.UserId
    Name = action.Name
    Email = action.Name
   }
  { this with Users = this.Users.Add(user.Id, user); }

Creating identifiers in stores

When creating a new object in a store based on an action, at times I forget and do thing like this:

member this.AddUser (action:AddUser)=
 let user = 
  {
   Id = UserId (Guid.NewGuid())
   Name = action.Name
   Email = action.Name
  }
 { this with Users = this.Users.Add(user.Id, user); }

The end result works the first time. But if you restart the service and Replay actions, then you will end up with a different Identifier for the User object in this case. All actions that want to use this object and refer to it by its Identifier, will not find a match after a replay. Not a good thing
A better way to write this would be:

type AddUser = { ActionId: Guid; UserId : UserId; Name:string; Email:string }
type UserStore =
 {
  Users: Map<UserId, User>
 }
 member this.AddUser (action:AddUser)=
  let user = 
   {
    Id = action.UserId
    Name = action.Name
    Email = action.Name
   }
  { this with Users = this.Users.Add(user.Id, user); }

This way, when we want to create a new user, the code that creates the AddUser action (maybe your ASP.NET Core MVC Controller, needs to generate the Id, but in turn the Id will be persisted with the action and will be the same even after a replay

Assuming that things exist in the store

Not all things are eternal. So just because there is an action arriving that thinks that there exists an object in the store does not mean that there actually is one. Another action could have removed it between that the first action was created and processed.

member this.ModifyUser (action:ModifyUser)=
    let original = this.Users.[action.UserId]
    let updated = { original with Name = action.Name; Email = action.Email }
    { this with Users = this.Users.Add(updated.Id, updated); }

The code above would throw an KeyNotFoundException if the key doesn't exist in the Map.
A better way to write this would be to try to find the object.

member this.ModifyUser (action:ModifyUser)=
 let o = this.Users.TryFind action.UserId
 match o with
 | None -> this
 | Some original ->
  let updated = { original with Name = action.Name; Email = action.Email }
  { this with Users = this.Users.Add(updated.Id, updated); }

Here we use the Map.TryFind function that returns None or Some x. If the key doesn't exist and we receive None, we jut return the current state as the new state.
If we got an hit, the Some original case would be hit and we can plug in the original code there.

Is it an event or a query?

I'm writing a game... Is each frame a new event that recalculates all positions of game objects... In the first version that was the case.

type ShipUpdateCoordinates =  { ActionId: Guid; Timestamp:DateTime; ShipId: ShipId; NewPosition:Vector3; }

Turns out that in 60 fps, there will be a lot of events persisted.

So. Instead of updating the position of each object in 60 fps. Then only event that is persisted is based on user input (or when the game engine decides to change the direction of an NPC object)

type ShipFlyTo =  { ActionId: Guid; Timestamp:DateTime; ShipId: ShipId; Origin:Vector3; Target:Vector3; }

The rendering engine will query

type TravelPlan =
 {
  Origin : Vector3
  Target : Vector3
  SpeedMetersPerSecond : float
  TravelStarted : DateTime
 }
 member this.Distance () = this.Origin.distance this.Target
 member this.TravelTimeSeconds () = 
  match this.SpeedMetersPerSecond with
  | 0. -> 0.
  | _ -> this.Distance() / this.SpeedMetersPerSecond
 member this.Eta () = 
  match this.TravelTimeSeconds() with
  | 0. -> DateTime.UtcNow
  | a -> this.TravelStarted.AddSeconds(a)

 member this.GetCoordinates (now:DateTime) =
  let eta = this.Eta()
  let percentage = PercentageOfValueBetweenMinAndMax (float this.TravelStarted.Ticks) (float eta.Ticks) (float now.Ticks)
  this.Origin.lerp this.Target (percentage/100.)

I.e. we know then the event occurred and what time it is now. That way we can calculate the current position of the object based on its speed and assuming it is traveling at constant speed by using linear interpolation (lerp)
The rendering engine can query the model as many times per second as it wants without having to store the new coordinates. As the ApplicationState is immutable and lock-free, we can do this without affecting event processing.

First time start?

I wanted to generate a game world on the first time start. I tried to use the processed events counter. Turns out as all processing is running on a different thread (as a MailboxProcessor), there is a race condition. And sometimes we just generate 2 worlds.. Not a good thing
Hence I updated the AppHolder class

let mutable private hasReplayed : bool = false;
let IsFirstTimeUse() = not hasReplayed

let private HandleReplayAction (action:obj, id:Guid) =
 hasReplayed <- true
 Processor.Post (Replay action)

This way, in my Program.cs that handles the startup I can just write this to ensure that the game world is generated on the first use.

AppHolder.InitiateFromLastSnapshot();
if (AppHolder.IsFirstTimeUse())
 GameBasicDataCreator.CreateOnFirstTimeStart();

When do I start the Game engine

A similar question, but took me too long to figure out it even happened.

AppHolder.InitiateFromLastSnapshot();
if (AppHolder.IsFirstTimeUse())
 GameBasicDataCreator.Create();
_engine.Start();

Here, we start the game engine when there is a race condition that Replay events are still being processed on another thread. The game engine will start analyzing the game state and reacting to things it see. I.e. not a good thing

Another change was needed in the AppHolder to fix this

let mutable private replayCompleted : bool = false;
let IsReplayCompleted() = replayCompleted
type Message =
 | Snapshot of AppliationState
 | Replay of obj
 | ReplayCompleted
 | Action of obj

For starters we add new public function so that we can query if the Replay is completed. We also add a new internal Message union case ReplayCompleted.

let private Processor = MailboxProcessor.Start(fun inbox ->
 let rec loop (s : AppliationState, c : int) =
  async {
   let! message = inbox.Receive()
   let c' = c + 1
   let s' =
    match message with
    | Snapshot snapshot -> snapshot
    | Replay a -> s.HandleAction a
    | ReplayCompleted ->
     replayCompleted <- true
     s
    | Action a -> 
     AppPersister.Persist a s c'
     s.HandleAction a
   state <- s'
   counter <- c'
   return! loop (s', c')
   }
 loop (AppliationState.Default, 0))

Next we change our MailboxProcessor code to include the handling for the new union case. We just set the mutable value to true from false.

let InitiateFromLastSnapshot () =
 let (snapshot, actions) = AppPersister.GetLatestSnapshotAndActions()
 match snapshot with
 | Some x -> Processor.Post (Snapshot x)
 | _ -> ()
 Array.map (fun x -> (HandleReplayAction x)) actions |> ignore
 if hasReplayed then Processor.Post ReplayCompleted
 
let InitiateFromActionsOnly () =
 AppPersister.GetAllActions()
 |> Array.map (fun x -> (HandleReplayAction x))
 |> ignore
 if hasReplayed then Processor.Post ReplayCompleted

And lastly, we change our 2 initialize functions to send in the ReplayCompleted message to the MailboxProcessor if there were any Replayed actions at all
In our initialization code in Program.cs we can now add some extra checks if there was any replaying done so that we actually wait for them all to be completed before continuing

AppHolder.InitiateFromLastSnapshot();
if (!AppHolder.IsFirstTimeUse())
{
 while (!AppHolder.IsReplayCompleted())
  Thread.Sleep(100);
}

That is it!
AppHolder updates have been uploaded to github.

All code provided as-is. This is copied from my own code-base, May need some additional programming to work. Use for whatever you want, how you want! If you find this helpful, please leave a comment or share a link, not required but appreciated! :)

Hope this helps someone out there!

Tuesday, June 5, 2018

Functional Adventures in F# - Type-safe identifiers

type ShipId = Guid
type Ship = 
 {
  Id : ShipId
  Name : string
 }
let ship = { Id : Guid.NewGuid(); Name : "Nebuchadnezzar" }

Turns out this was not the best way to do things. From C# code you can just send in any Guid, nothing is typed. And you can assign a 'ShipId' to a 'UserId' in F# as well, as it is just a type alias. I found out that a hard way while tracking down a bug that turned out to be a copy paste issue.

So, now that I have a bunch of stuff already written, I had to change this to a single item discriminated union. Took a while but it seems to work much better with the type-safety.. The way I thought that I thought it already did.

So, lets redefine the ShipId as a single item discriminated union like

type ShipId = ShipId of Guid

The code to use it from code is like:

let ship = { Id : ShipId (Guid.NewGuid()); Name : "Nebuchadnezzar" }

Usage from C#

My game is partially written in C# as it just works better for some tasks. So interfacing with the new typed ids will be like:

Getting the underlying Guid

var shipIdGuid = ship.Id.Item,

Creating a new ShipId from a Guid

var shipId = ShipId.NewShipId(Guid.NewGuid());

So a little bit more hassle from C#, but not that much, and most of my code is just service layer code creating F# records from external input. And the little extra overhead is worth it for the type-safety!

All code provided as-is. This is copied from my own code-base, May need some additional programming to work. Use for whatever you want, how you want! If you find this helpful, please leave a comment or share a link on social media, not required but appreciated! :)

Hope this helps someone out there!
Follow my preparation for adventure cycling through Europe!

Friday, May 18, 2018

Functional Adventures in F# - Adding Snapshot Persisting

In this post we will look at how to add snapshots to our event source engine from previous posts.

This post is part of a series:
Functional Adventures in F# - A simple planner
Functional Adventures in F# - Calling F# from C#
Functional Adventures in F# - Using Map Collections
Functional Adventures in F# - Application State
Functional Adventures in F# - Types with member functions
Functional Adventures in F# - Getting rid of loops
Functional Adventures in F# - Getting rid of temp variables
Functional Adventures in F# - The MailboxProcessor
Functional Adventures in F# - Persisting Application State
Functional Adventures in F# - Adding Snapshot Persisting
Functional Adventures in F# - Type-safe identifiers
Functional Adventures in F# - Event sourcing lessons learned

Quickest way to get started for me was to just persist a snapshot every X actions that were processed. Also for my needs, it is OK to persist the snapshot on a separate thread, as long as the actions are persisted by the AppHolder MailboxProcessor, then the snapshot can be written by another MailboxProcessor. This way, as the state grows it will not affect the main processing...

Changes needed to the AppPersister

We will be adding a new function called Persist that will be called from the AppHolder instead of the current PersistAction.

    let Persist (nextAction:obj) (state:obj) (actionCounter:int)  =
        PersistAction nextAction nextAction?ActionId
        PersistSnapshot actionCounter state nextAction?ActionId
        ()

This in turn will call 2 functions, the old PersistAction from previous part and the new PersistSnapshot.
Also, note the nextAction?ActionId, this is part of the FSharp.Interop.Dynamic NuGet package that adds the support for Dynamics, i.e. runtime checked fields and functions on objects instead of compile time checked types. We will be assuming that all actions (that are of type object, or obj in F#) implement the field ActionId... We could solve this by forcing an interface implementation but that just gives us ugly code here, but I guess there is a better solution but for me this solved the problem and I am only to blame myself if things break.
Use them with care.

    let private Snapshotter = MailboxProcessor.Start(fun inbox ->
        let rec loop (n:int) =
            async {
                let! (actionCounter:int,state:obj, nextActionId:Guid) = inbox.Receive()
                if actionCounter % 300 = 0 then
                    writeSnapshotToDisk state nextActionId
                return! loop n
                }
        loop 0)

    let PersistSnapshot (actionCounter:int) (state:obj) (nextActionId:Guid) =
        Snapshotter.Post (actionCounter, state, nextActionId)

The PersistSnapshot function just posts the state to the MailboxProcessor called Snapshotter that in turn calls writeSnahotToDisk every 300 action. Much hardcoded here, I guess you will have to experiment a little to find the magic number that works for you (300 is just an example and I will probably end changing it in my code as well)

    let private createSnapshotFilename (nextActionId:Guid) =
        let now = DateTime.UtcNow
        let dayPath = now.ToString("yyyy-MM-dd")
        let fullPath = Path.Combine(store, "snapshots", dayPath)
        let di = new DirectoryInfo(fullPath)
        di.Create()
        let filename = now.ToString("yyyy-MM-dd hh_mm_ss_fffffff.") + nextActionId.ToString() + ".snapshot"
        let fullFilename = Path.Combine(fullPath, filename)
        fullFilename
    
    let private writeSnapshotToDisk  (state:obj) (nextActionId:Guid) =
        let fullFilename = createSnapshotFilename nextActionId
        let json = JsonConvert.SerializeObject(state)
        File.WriteAllText(fullFilename,  json)
        ()

So basically as in the PersistAction part, we add some meta-data to the filename that we can use when reading from disk later. We JSON serialize the state object just as it is. The meta-data that we are interested in is the ID of the next action to be executed.

We will also be changing the createActionFilename function to add the ID of the action to the filename as follows:

    let private createActionFilename (action:obj) (actionId:Guid) =
        let now = DateTime.UtcNow
        let hourPath = now.ToString("yyyy-MM-dd HH")
        let fullPath = Path.Combine(store, hourPath)
        let di = new DirectoryInfo(fullPath)
        di.Create()
        let t = action.GetType()
        let filename = now.ToString("yyyy-MM-dd hh_mm_ss_fffffff+") + now.Ticks.ToString() + "." + t.Name + "." + actionId.ToString() +  ".action"
        let fullFilename = Path.Combine(fullPath, filename)
        fullFilename

Next we will be adding functionality to load snapshots and actions from disk. I.e. latest snapshot and all actions persisted after that snapshot. A totality that will result in a total state.

First, a minor change to getAction

    let private getAction (json:string, filename:string) =
        let split = filename.Split('.')
        let actionName = split.[1]
        let actionNameWithNamespace = "dreamstatecoding.core.Actions+" + actionName
        let t = Assembly.GetExecutingAssembly().GetType(actionNameWithNamespace)
        (JsonConvert.DeserializeObject(json, t), Guid.Parse(split.[2]))

We will be returning a tuple, with the deserialized object as the first element and the ID as the second element

Next, we want to get all actions from a specific ID, namely the ID that we wrote into the filename of the snapshot. To do this we do the following:

    let private fileContainAction (filename:string) (actionIdString:string) =
        let split = filename.Split('.')
        split.[2] = actionIdString

    let GetAllActionsFromId (nextActionId:Guid) =
        let di = new DirectoryInfo(store)
        di.Create()
        let nextActionIdString = nextActionId.ToString()
        let actions =
            di.GetFiles("*.action", SearchOption.AllDirectories)
            |> Seq.skipWhile (fun (fi:FileInfo) -> not (fileContainAction fi.Name nextActionIdString))
            |> Seq.map (fun (fi:FileInfo) -> File.ReadAllText(fi.FullName), fi.Name)
            |> Seq.map (fun x -> (getAction x))
            |> Seq.toArray
        actions

We use the Seq.skipWhile that skips elements in the input sequence until the function returns false, after that it will return the rest of the sequence. In our case, we check if the filename contains the next action id with the help of the fileContainsAction function and negating the result by using the not operator.
Handy when the framework contians these nice functions that do exactly what you need at the moment!

    let private getSnapshot (fi:FileInfo) =
        let split = fi.Name.Split('.')
        let json = File.ReadAllText(fi.FullName)
        (JsonConvert.DeserializeObject<ApplicationState.AppliationState>(json), Guid.Parse(split.[1]))

The getSnaphot is pretty much the same as the GetAction, we parse out the ID of the next action to be executed from the filename and return a tuple of the deserialized state and the ID.

And lastly the new function to be called from outside to get the latest state from disk storage:

    let GetLatestSnapshotAndActions () =
        let di = new DirectoryInfo(Path.Combine(store, "snapshots"))
        di.Create()
        let fileInfos = di.GetFiles("*.snapshot", SearchOption.AllDirectories)
        match fileInfos with
        | a when a.Length = 0 -> (None, GetAllActions())
        | _ ->
            let (snapshot, nextActionId) =
                fileInfos
                |> Seq.last
                |> getSnapshot
            let actions = GetAllActionsFromId nextActionId
            (Some snapshot, actions)

Key things to note here:

We create the snapshot directory if it does not exist, this so that we don't get 'directory does not exist' exceptions from the GetFiles method.
If there are no files returned by the GetFiles method, we return a tuple with None as the first element and the original GetAllActions as the actions list to be executed.
otherwie we get the last snapshot (Seq.last) and return Some snapshot, and reminder of actions as a tuple

The None and Some way of doing things seems nicer then using null as we do in other languages, especially from the caller side as we will see in our rewritten AppHolder module:

    type Message =
        | Snapshot of AppliationState
        | Replay of obj
        | Action of obj

First we introduce a new message type to our MailboxProcessor, the Snapshot of ApplicationState.

let s' =
 match message with
 | Snapshot snapshot -> snapshot
 | Replay a -> s.HandleAction a
 | Action a -> 
  AppPersister.Persist a s c'
  s.HandleAction a

If it is a snapshot, then the s' state is the snapshot, i.e. no other processing is done
Then, lastly we add a new init function to be called from our application startup code (Program.cs or whatever you are using)

    let InitiateFromLastSnapshot () =
        let (snapshot, actions) = AppPersister.GetLatestSnapshotAndActions()
        match snapshot with
        | Some x -> Processor.Post (Snapshot x)
        | _ -> ()
        Array.map (fun x -> (HandleReplayAction x)) actions

Here we get the snapshot and actions from the AppPersister, then we use pattern matching on the snapshot to determine if it is Some, in which case we send it into the Processor as a Snapshot message. Otherwise we do nothing ()
And then just run in all the Replay actions as before.

That is it!
All code is available at github.

Known issues

Newtonsoft JSON.NET typeconverter issue

Newtonsoft JSON.NET seems to have some trouble with F# Map collections, especially when using union types as keys

type UserId = UserId of Guid.
Map<UserId, User>

Gives following nasty exception when trying to deserialize it as a key in a Map

Newtonsoft.Json.JsonSerializationException
  HResult=0x80131500
  Message=Could not convert string 'UserId 8fa6d5a6-9500-4aac-9f3e-a5b918b81c46' to dictionary key type 'dreamstatecoding.core.Model+UserId'. Create a TypeConverter to convert from the string to the key type object. Path 'UserStore.Users['UserId 8fa6d5a6-9500-4aac-9f3e-a5b918b81c46']', line 1, position 69.
  Source=Newtonsoft.Json
  StackTrace:
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary(IDictionary dictionary, JsonReader reader, JsonDictionaryContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at dreamstatecoding.core.AppPersister.getSnapshot(FileInfo fi) in C:\tfs\apps\dreamstatecoding.core\AppPersister.fs:line 104
   at dreamstatecoding.core.AppPersister.GetLatestSnapshotAndActions() in C:\tfs\apps\dreamstatecoding.core\AppPersister.fs:line 113
   at dreamstatecoding.core.AppHolder.InitiateFromLastSnapshot() in C:\tfs\apps\dreamstatecoding.core\AppHolder.fs:line 47
   at test.web.Program.Main(String[] args) in C:\tfs\apps\test\test.web\Program.cs:line 14

Inner Exception 1:
JsonSerializationException: Error converting value "UserId 8fa6d5a6-9500-4aac-9f3e-a5b918b81c46" to type 'dreamstatecoding.core.Model+UserId'. Path 'UserStore.Users['UserId 8fa6d5a6-9500-4aac-9f3e-a5b918b81c46']', line 1, position 69.

Inner Exception 2:
ArgumentException: Could not cast or convert from System.String to dreamstatecoding.core.Model+UserId.

This is fixed and a new version is on github.. Thanks to the really nice #fsharp community on twitter for the help!
Code changes to include the Fable.JsonConverter

let converters =
 [ Fable.JsonConverter () :> JsonConverter ] |> List.toArray :> IList<JsonConverter>

let settings =
 JsonSerializerSettings (
  Converters = converters,
  Formatting = Formatting.Indented,
  NullValueHandling = NullValueHandling.Ignore)

And then just using it in serializing and deserializing as follows:

let json = JsonConvert.SerializeObject(state, settings)
let newState = JsonConvert.DeserializeObject<ApplicationState.ApplicationState>(json, settings)

All code provided as-is. This is copied from my own code-base, May need some additional programming to work. Use for whatever you want, how you want! If you find this helpful, please leave a comment, not required but appreciated! :)

Hope this helps someone out there!

Wednesday, May 9, 2018

Functional Adventures in F# - Persisting Application State

Storing all actions

The idea is pretty simple, we will store all actions sent to the AppHolder MailboxProcessor and whenever we want to re-initiate the application state, we just load all the actions from storage and run them through the processor again and we should end up with the same application state.

This has been described by other people, for example Martin Fowler, if you have the time then head over there and read his article about event sourcing.

For this we will create a module called AppPersister. It will handle the actual storing and retrieving of the events from disk. So it will have side-effects and thus not be purely functional.... So lets just throw all those concepts out the door. But, this is a great opportunity to show some nice features of F# that will help write this kind of code a little more ... functional?
For serialization I chose to use the json format, and serializer is the JSON.NET library by Newtonsoft that can be added to your project with NuGet.

Some code for starters:

module AppPersister =
    open System
    open System.Reflection
    open System.IO
    open Newtonsoft.Json
    
    let private store = @".\store\"

Basically we just create a new module that will handle the persisting of actions... Here we also define that the store should be a subdirectory for the application.

    let private createActionFilename (action:obj) =
        let now = DateTime.UtcNow
        let hourPath = now.ToString("yyyy-MM-dd HH")
        let fullPath = Path.Combine(store, hourPath)
        let di = new DirectoryInfo(fullPath)
        di.Create()
        let t = action.GetType()
        let filename = now.ToString("yyyy-MM-dd hh_mm_ss_fffffff+") + now.Ticks.ToString() + "." + t.Name + ".action"
        let fullFilename = Path.Combine(fullPath, filename)
        fullFilename

The createActionFilename function does just that, it generates a new unique filename for the action by combining todays date and time, throwing in the total number of ticks to ensure that we always have a fresh value and lastly adding the type of action and the extension '.action'. Here we also create the directory if it does not exist already for the current date and hour (UTC), the DirectoryInfo.Create method is safe to run on an already existing directory so we do no other checking ourselves.

    let PersistAction (action:obj) =
        let fullFilename = createActionFilename action
        let json = JsonConvert.SerializeObject(action)
        File.WriteAllText(fullFilename,  json)
        ()

The PersisAction function handles the actual writing of the action to disk. We call createActionFilename to get the full filename and then use JsonConvert to serialize the action to a json string. Lastly we write the file with the nice File.WriteAllText method.

Now that we can write the actions, lets write some code to read them from disk.

    let GetAllActions () =
        let di = new DirectoryInfo(store)
        let actions =
            di.GetFiles("*.action", SearchOption.AllDirectories)
            |> Seq.map (fun (fi:FileInfo) -> File.ReadAllText(fi.FullName), fi.Name)
            |> Seq.map (fun x -> (getAction x))
            |> Seq.toArray
        actions

Here we use the DirectoryInfo.GetFiles built in method to find all files with the .action extension in any subdirectory in the store path. The result is an array of FileInfo objects that we pipe to a
Seq.map where we return a Tuple containing the file contents and filename and pipe that tuple into
another Seq.map where we call getAction for all elements. This function will be tasked with deserializing the json to the correct type
Lastly we pipe the contents to Seq.toArray to make the result concrete. To my understanding is that the Seq constructs work a little like the IEnumerable and do lazy evaluation if you do not actually list it.

    let private getAction (json:string, filename:string) =
        let split = filename.Split('.')
        let actionName = split.[1]
        let actionNameWithNamespace = "dreamstatecoding.core.Actions+" + actionName
        let t = Assembly.GetExecutingAssembly().GetType(actionNameWithNamespace)
        JsonConvert.DeserializeObject(json, t)

Lastly, the getAction function that is called from GetAllActions. Here we split the filename and pick out the part containing the name of the Action and then as the currently executing assembly to find the Type for that. Notice that we need to prefix the action name with the namespace name to make it work. In my solution I have all my actions in 1 module, so this works.

Rewriting AppHolder to work with persisted actions

Next step is to rewrite the AppHolder module from previous part to work with the AppPersister module.

So what we want to do here is to separate actual real time Actions from Replay actions from the persisting, mainly so that they do not get persisted again (duplicated).

So, lets put up a new discriminated union that defines what we want to do

    type Message =
        | Replay of obj
        | Action of obj

So, either we want to execute a Replay object or execute an Action object.

    let private Processor = Agent.Start(fun inbox ->
        let rec loop (s : AppliationState) =
            async {
                let! message = inbox.Receive()
                let (action:obj) = 
                    match message with
                    | Action a -> 
                        AppPersister.PersistAction a
                        a
                    | Replay a -> a

                let s' = s.HandleAction action
                state <- s'
                counter <- counter + 1
                return! loop s'
                }
        loop AppliationState.Default)

Now we rewrite our MailboxProcessor (Agent) to take a message instead of object directly. The first step is to pattern-match the received message to the type and get out the payload object.
If we are handling an Action we want to persist it as well, so lets put that function call here as well. Otherwise no change here.

    let HandleAction (action:obj) =
        Processor.Post (Action action)

The old HandleAction code called Processor.Post with the action directly. Now we must state that it is an Action, and not a Replay.

    let private HandleReplayAction (action:obj) =
        Processor.Post (Replay action)

The same goes for the Replay execution, we just add a new function that executes actions as Replays

    let InitiateFromStore () =
        AppPersister.GetAllActions()
        |> Array.map (fun x -> (HandleReplayAction x))
        |> ignore
        ()

Lastly, a function to initialize the store, we pipe all the actions loaded from persistence to the HandleReplayAction and then pipe the results from that to ignore. This means that we are not really interested in the results from this function call, in the end we just return unit from the InitiateFromStore function.
I put the call to this into my Program.cs file, it will load all the actions already executed from the store and execute them all as Replay actions.

Conclusions

In this part we have looked at how to persist the application state as a series of actions and then re-creating the application state when the application is restarted.
For applications handling lots and lots of actions, we may want to limit the number of actions needed to recreate the up to date state.. So in the next part we will look at snapshots. My plan is to upload the code to git after the snapshot part has been added.

All code provided as-is. This is copied from my own code-base, May need some additional programming to work. Use for whatever you want, how you want! If you find this helpful, please leave a comment, not required but appreciated! :)

Hope this helps someone out there!

Saturday, April 28, 2018

Functional Adventures in F# - The MailboxProcessor

MailboxProcessor

This continues with the code from Application State post.
Mainly the GameState type

and GameState =
 {
  PlayerStore:PlayerStore
  UnitStore:UnitStore
 }
 member this.HandleAction (action:obj) (state:GameState) =
  {
   PlayerStore = state.PlayerStore.HandleAction action state.UserStore
   UnitStore = state.UnitStore.HandleAction action state.AmmunitionStore 
  }

My goal is to incorporate F# with C# in a nice way. And as described in the Application State post, the core application model and all actions that can be made in the application will be modeled in F#. But to prevent a multithreaded C# application to corrupt the state we need to serialize the commands that are processed.
In C# I would have used a background Thread and a Queue<T> and written a locks to synchronize the reads and writes. Luckily in F#, we have all that nicely packaged in the built in MailboxProcessor.

The first line redefines the MailboxProcessor to the term Agent instead, this seems to be something that a lot of people are doing, so lets stick to that for the time being.
After that we define a F# static class, i.e. an abstract and sealed class with only static members. (this is what C# does under the hood).

type Agent<'T> = MailboxProcessor<'T>     
[<AbstractClass; Sealed>]
type GameHolder private () =
 static let mutable state : GameState = 
  {
   PlayerStore = { Players = Map.empty }
   UnitStore = { Units = Map.empty }
  }
 static member State : GameState = state
 static member private Processor = Agent.Start(fun inbox ->
  let rec loop =
   async {
    let! action = inbox.Receive()
    state <- state.HandleAction action state
    return! loop
    }
  loop)
 static member HandleAction (action:obj) =
  GameHolder.Processor.Post action

In addition we will be allowing for mutation here, something that the business logic in the F# module should be unaware of, but here, we are exposing the current state to the rest of the application so that it is easily accessible.

I.e. C# we can call GameHolder.State and always get the latest state. It is exposed as immutable so there is no risk that other threads will write to it.

After that we define the MailboxProcessor<'T> or in this case the redefined Agent<'T> as a private static member of the class.
What happens in the recursive part is a little iffy for me still. To my understanding the let! (with the exclamation mark) makes the inbox.Receive (that is also an async call) an synchronous call and stores its result in the action value.
If anyone has detailed explanation of what is going on here, please feel free to comment as I was unable to find details of this seems just to be the way things are done.

The HandleAction method calls the Post method on the MailboxProcessor and thus posts the action to the MailboxProcessor that in turn calls the HandleAction method of the State and mutates the exposed State value.

Update 2018-05-02, forcing serial access

Turns out the MailboxProcessor above does not work the way I thought... In my mind it serialized the processing but it did not. I.e. when multiple actions were sent to it in parallel, it corrupted the state variable.
After looking at this for some time now, I finally had to post the question on stackoverflow.
https://stackoverflow.com/questions/50135825/f-mailboxprocessor-limit-parallelism

Turns out the bad wolf in the drama was the static member. Changing the code to a module instead got the solution to work as expected.

The end result is:

module GameHolder =
    type Agent<'T> = MailboxProcessor<'T>     

    let mutable private state = AppliationState.Default
    let mutable private counter : int = 0
    let GetCurrentState() =  state
    let GetProcessedActionsCounter() = counter
    let private Processor = Agent.Start(fun inbox ->
        let rec loop (s : AppliationState) =
            async {
                let! action = inbox.Receive()
                let s' = s.HandleAction action
                state <- s'
                counter <- counter + 1
                return! loop s'
                }
        loop AppliationState.Default)

    let HandleAction (action:obj) =
        Processor.Post action

I can still get the current application state by calling a 'static' method from C#, in this case GameHolder.GetCurrentState().
So a small change thanks to stackoverflow and we still have a small but neat interface between our C# game/application and it model/business logic that is written in F#.

Finally

In the end. We have a GameHolder class that exposes State and HandleAction to the outside world. Whoever can query the state at anytime, but they are not allowed to change it other then through the HandleAction method.

All code provided as-is. This is copied from my own code-base, May need some additional programming to work. Use for whatever you want, how you want! If you find this helpful, please leave a comment or share a link, not required but appreciated! :)

Hope this helps someone out there!

Thursday, March 22, 2018

Functional Adventures in F# - Getting rid of temp variables

static member HandleTick (state:UnitStore) (action:Tick) =     
 let queueUpdatedState = UnitStore.handleTickBuildQueues state action
 let unitsUpdatedState = UnitStore.handleTickUnitUpdates queueUpdatedState action
 unitsUpdatedState

A function that takes a state and an action and in turn it uses other functions to update that state with the help of the action.
I don't know, but I grew tired of naming temporary variables after a while and started googling for a better solution. Turns out there is one. By using the forward-pipe operator '|>' we can take the output of one function and send it as input to the next function.

static member HandleTick (state:UnitStore) (action:Tick) =     
 UnitStore.handleTickBuildQueues action state 
 |> UnitStore.handleTickUnitUpdates action

To make this work, the order of parameters need to be switched so that Action is the first argument followed by the State.
Turns out I have to refactor all the functions in my Stores to take the action first and state as second argument to get this working, but I guess that is a small price to pay for cleaner code.

Looking at the assembly in ILSpy gives us that in the background there will be temporary variables passed around, but our code is clean of them.

public static UnitStore HandleTick(Actions.Tick action, UnitStore state)
{
 UnitStore unitStore = UnitStore.handleTickBuildQueues(action, state);
 UnitStore state2 = unitStore;
 return UnitStore.handleTickUnitUpdates(action, state2);
}

All code provided as-is. This is copied from my own code-base, May need some additional programming to work. Use for whatever you want, how you want! If you find this helpful, please leave a comment, not required but appreciated! :)

Hope this helps someone out there!

Wednesday, March 21, 2018

Functional Adventures in F# - Getting rid of loops

private Ship FindClosestFreeCoordinate(ImmutableList<Fleet> state, Ship newShip)
{
    var shipsWithinSafetyDistance = ShipsWithinSafetyDistance(state, newShip);
    while (shipsWithinSafetyDistance.Any())
    {
        var newCoordinates = newShip.Coordinates.Position.Add(Vector3.UnitXVector);
        newShip = newShip.Modify(coordinates: newShip.Coordinates.Modify(position: newCoordinates));
        shipsWithinSafetyDistance = ShipsWithinSafetyDistance(state, newShip);
    }
    return newShip;
}

private List<Ship> ShipsWithinSafetyDistance(ImmutableList<Fleet> state, Ship newShip)
{
    return (from fleet in state
        from
            ship in fleet.Ships
        where
            ship.Distance(newShip) < 5
        select ship).ToList();
}

Translating the ShipsWithinSafetyDistance function is no problem. We just use the Map.filter function...

static member private shipsWithinSafetyDistance (unit:Unit) (state:UnitStore) =
    Map.filter (fun key value -> unit.distanceToUnit value < 5.0) state.Units

Ah, yes.. I switched from a ImmutableList in C# to a Map in F#, just to keep things interesting (and to get faster Key lookups)

So, the tricky part here is the while loop. Most loops can be rewritten with Map/List functions like map, filter and fold etc... But this time I don't see how to use them as I don't have a Collection of any sort here so my brain keeps yelling "Write a loooooop", as that is the way I have solved this kind of problems for the past 20+ years... So...
How should we do it in F#??? We have some other constructs to use that fit the functional idea better.

static member private findClosestFreeCoordinate (unit:Unit) (state:UnitStore) =
 let rec findClosestFreeCoordinateRecursive (unit:Unit) (state:UnitStore) = 
  let shipsWithinSafetyDistance = shipsWithinSafetyDistance unit state
  match shipsWithinSafetyDistance with
  | a when a.IsEmpty -> unit
  | _ ->  let newUnit = { unit with Coordinates = unit.Coordinates + Vector3.UnitX }
    findClosestFreeCoordinateRecursive newUnit state
 findClosestFreeCoordinateRecursive unit state

Recursion!
At least for this problem, it seems to be a good option. Here we define a function, findClosestFreeCoordinateRecursive with the rec keyword to let the compiler know that it will be called recursively. In the function body we match the result list with IsEmpty and return the input value or else call the recursive function with a new copy of the unit with new coordinates.

Call stack.. Is there a risk for stack overflow with this?
Lets open the dll output with ILSpy to check what actually goes on behind the scenes

Turns out, that the current F# compiler in the background uses a while loop. The key here is that we do not, instead we define a short and concise function where we state what it is that we want done, how it is solved behind the scene in detail is not up to us. At some point, there is always someone that needs to write the ifs and loops, but lets keep them away from the application code that we write.

So there, lets move the last shipsWithinSafetyDistance to inner scope of our function and we are set

static member private findClosestFreeCoordinate (unit:Unit) (state:UnitStore) =
 let shipsWithinSafetyDistance (unit:Unit) (state:UnitStore) =
  Map.filter (fun _ value -> unit.distanceToUnit value < 50.0) state.Units
 let rec findClosestFreeCoordinateRecursive (unit:Unit) (state:UnitStore) = 
  let shipsWithinSafetyDistance = shipsWithinSafetyDistance unit state
  match shipsWithinSafetyDistance with
  | a when a.IsEmpty -> unit
  | _ ->  let newUnit = { unit with Coordinates = unit.Coordinates + Vector3.UnitX }
    findClosestFreeCoordinateRecursive newUnit state
 findClosestFreeCoordinateRecursive unit state

All code provided as-is. This is copied from my own code-base, May need some additional programming to work. Use for whatever you want, how you want! If you find this helpful, please leave a comment, not required but appreciated! :)

Hope this helps someone out there!

Pages

Saturday, December 8, 2018

Wednesday, October 17, 2018

Getting started

Configure script editor

Logging to screen

Logger script

Saturday, September 29, 2018

First off, what are immutable objects

Why bother?

What options do we have in .NET?

Readonly objects in C# with constructor lists

Readonly objets in C# with Modify pattern

Record types in F#

Performance

Performance test source code

Results

Rewritten increment example

Conclusions

Friday, September 28, 2018

Wednesday, September 26, 2018

Wednesday, June 20, 2018

Using domain objects in actions

Creating identifiers in stores

Assuming that things exist in the store

Is it an event or a query?

First time start?

When do I start the Game engine

Tuesday, June 5, 2018

Usage from C#

Friday, May 18, 2018

Known issues

Newtonsoft JSON.NET typeconverter issue

Wednesday, May 9, 2018

Storing all actions

Rewriting AppHolder to work with persisted actions

Conclusions

Saturday, April 28, 2018

MailboxProcessor

Update 2018-05-02, forcing serial access

Finally

Thursday, March 22, 2018

Wednesday, March 21, 2018