Wednesday, June 20, 2018

Functional Adventures in F# - Event sourcing lessons learned

6 things I wish I knew before starting with event sourcing

Thought I'd write something about the pitfalls that I've stumbled across when using my event sourcing engine.

This post is part of a series:
Functional Adventures in F# - A simple planner
Functional Adventures in F# - Calling F# from C#
Functional Adventures in F# - Using Map Collections
Functional Adventures in F# - Application State
Functional Adventures in F# - Types with member functions
Functional Adventures in F# - Getting rid of loops
Functional Adventures in F# - Getting rid of temp variables
Functional Adventures in F# - The MailboxProcessor
Functional Adventures in F# - Persisting Application State
Functional Adventures in F# - Adding Snapshot Persisting
Functional Adventures in F# - Type-safe identifiers
Functional Adventures in F# - Event sourcing lessons learned

Using domain objects in actions

The first actions that I used basically had me create the domain objects on the outside and then plug them into the create action. The store only took the object and placed it in the stores Map over objects.
type AddUser = { ActionId: Guid; User : User }
type UserStore =
 {
  Users: Map<UserId, User>
 }
 member this.AddUser (action:AddUser)=
  { this with Users = this.Users.Add(action.User.Id, action.User); }

 member this.HandleAction (action:obj) =
  match action with
  | :? AddUser as a -> this.AddUser a
  | _ -> this 
This makes your domain model a part of the persistence layer (as all actions are persisted). I.e. you cannot change your model without getting nifty errors (like null exceptions)
You might think that: You still persist snapshots of the whole ApplicationState, i.e. the User object ends up in persistence.
True. But when you do 'model breaking changes', you could just remove all snapshots and then rerun all actions from the start and get the equivalent state as you did before the breaking change. If you in turn include the domain object in the actions, you are screwed.
A better way to write this would be:
type AddUser = { ActionId: Guid; UserId : UserId; Name:string; Email:string }
type UserStore =
 {
  Users: Map<UserId, User>
 }
 member this.AddUser (action:AddUser)=
  let user = 
   {
    Id = action.UserId
    Name = action.Name
    Email = action.Name
   }
  { this with Users = this.Users.Add(user.Id, user); }



Creating identifiers in stores

When creating a new object in a store based on an action, at times I forget and do thing like this:
member this.AddUser (action:AddUser)=
 let user = 
  {
   Id = UserId (Guid.NewGuid())
   Name = action.Name
   Email = action.Name
  }
 { this with Users = this.Users.Add(user.Id, user); }
The end result works the first time. But if you restart the service and Replay actions, then you will end up with a different Identifier for the User object in this case. All actions that want to use this object and refer to it by its Identifier, will not find a match after a replay. Not a good thing
A better way to write this would be:
type AddUser = { ActionId: Guid; UserId : UserId; Name:string; Email:string }
type UserStore =
 {
  Users: Map<UserId, User>
 }
 member this.AddUser (action:AddUser)=
  let user = 
   {
    Id = action.UserId
    Name = action.Name
    Email = action.Name
   }
  { this with Users = this.Users.Add(user.Id, user); }
This way, when we want to create a new user, the code that creates the AddUser action (maybe your ASP.NET Core MVC Controller, needs to generate the Id, but in turn the Id will be persisted with the action and will be the same even after a replay

Assuming that things exist in the store

Not all things are eternal. So just because there is an action arriving that thinks that there exists an object in the store does not mean that there actually is one. Another action could have removed it between that the first action was created and processed.
member this.ModifyUser (action:ModifyUser)=
    let original = this.Users.[action.UserId]
    let updated = { original with Name = action.Name; Email = action.Email }
    { this with Users = this.Users.Add(updated.Id, updated); }
The code above would throw an KeyNotFoundException if the key doesn't exist in the Map.
A better way to write this would be to try to find the object.
member this.ModifyUser (action:ModifyUser)=
 let o = this.Users.TryFind action.UserId
 match o with
 | None -> this
 | Some original ->
  let updated = { original with Name = action.Name; Email = action.Email }
  { this with Users = this.Users.Add(updated.Id, updated); }
Here we use the Map.TryFind function that returns None or Some x. If the key doesn't exist and we receive None, we jut return the current state as the new state.
If we got an hit, the Some original case would be hit and we can plug in the original code there.

Is it an event or a query?

I'm writing a game... Is each frame a new event that recalculates all positions of game objects... In the first version that was the case.
type ShipUpdateCoordinates =  { ActionId: Guid; Timestamp:DateTime; ShipId: ShipId; NewPosition:Vector3; }
Turns out that in 60 fps, there will be a lot of events persisted.

So. Instead of updating the position of each object in 60 fps. Then only event that is persisted is based on user input (or when the game engine decides to change the direction of an NPC object)
type ShipFlyTo =  { ActionId: Guid; Timestamp:DateTime; ShipId: ShipId; Origin:Vector3; Target:Vector3; }
The rendering engine will query
type TravelPlan =
 {
  Origin : Vector3
  Target : Vector3
  SpeedMetersPerSecond : float
  TravelStarted : DateTime
 }
 member this.Distance () = this.Origin.distance this.Target
 member this.TravelTimeSeconds () = 
  match this.SpeedMetersPerSecond with
  | 0. -> 0.
  | _ -> this.Distance() / this.SpeedMetersPerSecond
 member this.Eta () = 
  match this.TravelTimeSeconds() with
  | 0. -> DateTime.UtcNow
  | a -> this.TravelStarted.AddSeconds(a)

 member this.GetCoordinates (now:DateTime) =
  let eta = this.Eta()
  let percentage = PercentageOfValueBetweenMinAndMax (float this.TravelStarted.Ticks) (float eta.Ticks) (float now.Ticks)
  this.Origin.lerp this.Target (percentage/100.)
I.e. we know then the event occurred and what time it is now. That way we can calculate the current position of the object based on its speed and assuming it is traveling at constant speed by using linear interpolation (lerp)
The rendering engine can query the model as many times per second as it wants without having to store the new coordinates. As the ApplicationState is immutable and lock-free, we can do this without affecting event processing.

First time start?

I wanted to generate a game world on the first time start. I tried to use the processed events counter. Turns out as all processing is running on a different thread (as a MailboxProcessor), there is a race condition. And sometimes we just generate 2 worlds.. Not a good thing
Hence I updated the AppHolder class
let mutable private hasReplayed : bool = false;
let IsFirstTimeUse() = not hasReplayed

let private HandleReplayAction (action:obj, id:Guid) =
 hasReplayed <- true
 Processor.Post (Replay action)
This way, in my Program.cs that handles the startup I can just write this to ensure that the game world is generated on the first use.
AppHolder.InitiateFromLastSnapshot();
if (AppHolder.IsFirstTimeUse())
 GameBasicDataCreator.CreateOnFirstTimeStart();

When do I start the Game engine

A similar question, but took me too long to figure out it even happened.
AppHolder.InitiateFromLastSnapshot();
if (AppHolder.IsFirstTimeUse())
 GameBasicDataCreator.Create();
_engine.Start();
Here, we start the game engine when there is a race condition that Replay events are still being processed on another thread. The game engine will start analyzing the game state and reacting to things it see. I.e. not a good thing

Another change was needed in the AppHolder to fix this
let mutable private replayCompleted : bool = false;
let IsReplayCompleted() = replayCompleted
type Message =
 | Snapshot of AppliationState
 | Replay of obj
 | ReplayCompleted
 | Action of obj
For starters we add new public function so that we can query if the Replay is completed. We also add a new internal Message union case ReplayCompleted.
let private Processor = MailboxProcessor.Start(fun inbox ->
 let rec loop (s : AppliationState, c : int) =
  async {
   let! message = inbox.Receive()
   let c' = c + 1
   let s' =
    match message with
    | Snapshot snapshot -> snapshot
    | Replay a -> s.HandleAction a
    | ReplayCompleted ->
     replayCompleted <- true
     s
    | Action a -> 
     AppPersister.Persist a s c'
     s.HandleAction a
   state <- s'
   counter <- c'
   return! loop (s', c')
   }
 loop (AppliationState.Default, 0))
Next we change our MailboxProcessor code to include the handling for the new union case. We just set the mutable value to true from false.
let InitiateFromLastSnapshot () =
 let (snapshot, actions) = AppPersister.GetLatestSnapshotAndActions()
 match snapshot with
 | Some x -> Processor.Post (Snapshot x)
 | _ -> ()
 Array.map (fun x -> (HandleReplayAction x)) actions |> ignore
 if hasReplayed then Processor.Post ReplayCompleted
 
let InitiateFromActionsOnly () =
 AppPersister.GetAllActions()
 |> Array.map (fun x -> (HandleReplayAction x))
 |> ignore
 if hasReplayed then Processor.Post ReplayCompleted
And lastly, we change our 2 initialize functions to send in the ReplayCompleted message to the MailboxProcessor if there were any Replayed actions at all
In our initialization code in Program.cs we can now add some extra checks if there was any replaying done so that we actually wait for them all to be completed before continuing
AppHolder.InitiateFromLastSnapshot();
if (!AppHolder.IsFirstTimeUse())
{
 while (!AppHolder.IsReplayCompleted())
  Thread.Sleep(100);
}


That is it!
AppHolder updates have been uploaded to github.

All code provided as-is. This is copied from my own code-base, May need some additional programming to work. Use for whatever you want, how you want! If you find this helpful, please leave a comment or share a link, not required but appreciated! :)

Hope this helps someone out there!
Until next time: Work to Live, Don’t Live to Work

Tuesday, June 5, 2018

Functional Adventures in F# - Type-safe identifiers


In this post we will look at type-safe identifiers for things.

This post is part of a series:
Functional Adventures in F# - A simple planner
Functional Adventures in F# - Calling F# from C#
Functional Adventures in F# - Using Map Collections
Functional Adventures in F# - Application State
Functional Adventures in F# - Types with member functions
Functional Adventures in F# - Getting rid of loops
Functional Adventures in F# - Getting rid of temp variables
Functional Adventures in F# - The MailboxProcessor
Functional Adventures in F# - Persisting Application State
Functional Adventures in F# - Adding Snapshot Persisting
Functional Adventures in F# - Type-safe identifiers
Functional Adventures in F# - Event sourcing lessons learned

So, working on my game I just used to define identifiers like for example ShipId like the following:
type ShipId = Guid
type Ship = 
 {
  Id : ShipId
  Name : string
 }
let ship = { Id : Guid.NewGuid(); Name : "Nebuchadnezzar" }
Turns out this was not the best way to do things. From C# code you can just send in any Guid, nothing is typed. And you can assign a 'ShipId' to a 'UserId' in F# as well, as it is just a type alias. I found out that a hard way while tracking down a bug that turned out to be a copy paste issue.

So, now that I have a bunch of stuff already written, I had to change this to a single item discriminated union. Took a while but it seems to work much better with the type-safety.. The way I thought that I thought it already did.

So, lets redefine the ShipId as a single item discriminated union like
type ShipId = ShipId of Guid
The code to use it from code is like:
let ship = { Id : ShipId (Guid.NewGuid()); Name : "Nebuchadnezzar" }


Usage from C#

My game is partially written in C# as it just works better for some tasks. So interfacing with the new typed ids will be like:

Getting the underlying Guid
var shipIdGuid = ship.Id.Item,
Creating a new ShipId from a Guid
var shipId = ShipId.NewShipId(Guid.NewGuid());

So a little bit more hassle from C#, but not that much, and most of my code is just service layer code creating F# records from external input. And the little extra overhead is worth it for the type-safety!

All code provided as-is. This is copied from my own code-base, May need some additional programming to work. Use for whatever you want, how you want! If you find this helpful, please leave a comment or share a link, not required but appreciated! :)

Hope this helps someone out there!
Until next time: Work to Live, Don’t Live to Work


Friday, May 18, 2018

Functional Adventures in F# - Adding Snapshot Persisting



In this post we will look at how to add snapshots to our event source engine from previous posts.

This post is part of a series:
Functional Adventures in F# - A simple planner
Functional Adventures in F# - Calling F# from C#
Functional Adventures in F# - Using Map Collections
Functional Adventures in F# - Application State
Functional Adventures in F# - Types with member functions
Functional Adventures in F# - Getting rid of loops
Functional Adventures in F# - Getting rid of temp variables
Functional Adventures in F# - The MailboxProcessor
Functional Adventures in F# - Persisting Application State
Functional Adventures in F# - Adding Snapshot Persisting
Functional Adventures in F# - Type-safe identifiers
Functional Adventures in F# - Event sourcing lessons learned


Quickest way to get started for me was to just persist a snapshot every X actions that were processed. Also for my needs, it is OK to persist the snapshot on a separate thread, as long as the actions are persisted by the AppHolder MailboxProcessor, then the snapshot can be written by another MailboxProcessor. This way, as the state grows it will not affect the main processing...

Changes needed to the AppPersister

We will be adding a new function called Persist that will be called from the AppHolder instead of the current PersistAction.
    let Persist (nextAction:obj) (state:obj) (actionCounter:int)  =
        PersistAction nextAction nextAction?ActionId
        PersistSnapshot actionCounter state nextAction?ActionId
        ()
This in turn will call 2 functions, the old PersistAction from previous part and the new PersistSnapshot.
Also, note the nextAction?ActionId, this is part of the FSharp.Interop.Dynamic NuGet package that adds the support for Dynamics, i.e. runtime checked fields and functions on objects instead of compile time checked types. We will be assuming that all actions (that are of type object, or obj in F#) implement the field ActionId... We could solve this by forcing an interface implementation but that just gives us ugly code here, but I guess there is a better solution but for me this solved the problem and I am only to blame myself if things break.
Use them with care.

    let private Snapshotter = MailboxProcessor.Start(fun inbox ->
        let rec loop (n:int) =
            async {
                let! (actionCounter:int,state:obj, nextActionId:Guid) = inbox.Receive()
                if actionCounter % 300 = 0 then
                    writeSnapshotToDisk state nextActionId
                return! loop n
                }
        loop 0)

    let PersistSnapshot (actionCounter:int) (state:obj) (nextActionId:Guid) =
        Snapshotter.Post (actionCounter, state, nextActionId)

The PersistSnapshot function just posts the state to the MailboxProcessor called Snapshotter that in turn calls writeSnahotToDisk every 300 action. Much hardcoded here, I guess you will have to experiment a little to find the magic number that works for you (300 is just an example and I will probably end changing it in my code as well)
    let private createSnapshotFilename (nextActionId:Guid) =
        let now = DateTime.UtcNow
        let dayPath = now.ToString("yyyy-MM-dd")
        let fullPath = Path.Combine(store, "snapshots", dayPath)
        let di = new DirectoryInfo(fullPath)
        di.Create()
        let filename = now.ToString("yyyy-MM-dd hh_mm_ss_fffffff.") + nextActionId.ToString() + ".snapshot"
        let fullFilename = Path.Combine(fullPath, filename)
        fullFilename
    
    let private writeSnapshotToDisk  (state:obj) (nextActionId:Guid) =
        let fullFilename = createSnapshotFilename nextActionId
        let json = JsonConvert.SerializeObject(state)
        File.WriteAllText(fullFilename,  json)
        ()

So basically as in the PersistAction part, we add some meta-data to the filename that we can use when reading from disk later. We JSON serialize the state object just as it is. The meta-data that we are interested in is the ID of the next action to be executed.


We will also be changing the createActionFilename function to add the ID of the action to the filename as follows:
    let private createActionFilename (action:obj) (actionId:Guid) =
        let now = DateTime.UtcNow
        let hourPath = now.ToString("yyyy-MM-dd HH")
        let fullPath = Path.Combine(store, hourPath)
        let di = new DirectoryInfo(fullPath)
        di.Create()
        let t = action.GetType()
        let filename = now.ToString("yyyy-MM-dd hh_mm_ss_fffffff+") + now.Ticks.ToString() + "." + t.Name + "." + actionId.ToString() +  ".action"
        let fullFilename = Path.Combine(fullPath, filename)
        fullFilename


Next we will be adding functionality to load snapshots and actions from disk. I.e. latest snapshot and all actions persisted after that snapshot. A totality that will result in a total state.

First, a minor change to getAction
    let private getAction (json:string, filename:string) =
        let split = filename.Split('.')
        let actionName = split.[1]
        let actionNameWithNamespace = "dreamstatecoding.core.Actions+" + actionName
        let t = Assembly.GetExecutingAssembly().GetType(actionNameWithNamespace)
        (JsonConvert.DeserializeObject(json, t), Guid.Parse(split.[2]))
We will be returning a tuple, with the deserialized object as the first element and the ID as the second element

Next, we want to get all actions from a specific ID, namely the ID that we wrote into the filename of the snapshot. To do this we do the following:
    let private fileContainAction (filename:string) (actionIdString:string) =
        let split = filename.Split('.')
        split.[2] = actionIdString

    let GetAllActionsFromId (nextActionId:Guid) =
        let di = new DirectoryInfo(store)
        di.Create()
        let nextActionIdString = nextActionId.ToString()
        let actions =
            di.GetFiles("*.action", SearchOption.AllDirectories)
            |> Seq.skipWhile (fun (fi:FileInfo) -> not (fileContainAction fi.Name nextActionIdString))
            |> Seq.map (fun (fi:FileInfo) -> File.ReadAllText(fi.FullName), fi.Name)
            |> Seq.map (fun x -> (getAction x))
            |> Seq.toArray
        actions
We use the Seq.skipWhile that skips elements in the input sequence until the function returns false, after that it will return the rest of the sequence. In our case, we check if the filename contains the next action id with the help of the fileContainsAction function and negating the result by using the not operator.
Handy when the framework contians these nice functions that do exactly what you need at the moment!
    let private getSnapshot (fi:FileInfo) =
        let split = fi.Name.Split('.')
        let json = File.ReadAllText(fi.FullName)
        (JsonConvert.DeserializeObject<ApplicationState.AppliationState>(json), Guid.Parse(split.[1]))

The getSnaphot is pretty much the same as the GetAction, we parse out the ID of the next action to be executed from the filename and return a tuple of the deserialized state and the ID.

And lastly the new function to be called from outside to get the latest state from disk storage:
    let GetLatestSnapshotAndActions () =
        let di = new DirectoryInfo(Path.Combine(store, "snapshots"))
        di.Create()
        let fileInfos = di.GetFiles("*.snapshot", SearchOption.AllDirectories)
        match fileInfos with
        | a when a.Length = 0 -> (None, GetAllActions())
        | _ ->
            let (snapshot, nextActionId) =
                fileInfos
                |> Seq.last
                |> getSnapshot
            let actions = GetAllActionsFromId nextActionId
            (Some snapshot, actions)
Key things to note here:

  • We create the snapshot directory if it does not exist, this so that we don't get 'directory does not exist' exceptions from the GetFiles method.
  • If there are no files returned by the GetFiles method, we return a tuple with None as the first element and the original GetAllActions as the actions list to be executed.
  • otherwie we get the last snapshot (Seq.last) and return Some snapshot, and reminder of actions as a tuple
The None and Some way of doing things seems nicer then using null as we do in other languages, especially from the caller side as we will see in our rewritten AppHolder module:

    type Message =
        | Snapshot of AppliationState
        | Replay of obj
        | Action of obj
First we introduce a new message type to our MailboxProcessor, the Snapshot of ApplicationState.
let s' =
 match message with
 | Snapshot snapshot -> snapshot
 | Replay a -> s.HandleAction a
 | Action a -> 
  AppPersister.Persist a s c'
  s.HandleAction a
If it is a snapshot, then the s' state is the snapshot, i.e. no other processing is done
Then, lastly we add a new init function to be called from our application startup code (Program.cs or whatever you are using)

    let InitiateFromLastSnapshot () =
        let (snapshot, actions) = AppPersister.GetLatestSnapshotAndActions()
        match snapshot with
        | Some x -> Processor.Post (Snapshot x)
        | _ -> ()
        Array.map (fun x -> (HandleReplayAction x)) actions
Here we get the snapshot and actions from the AppPersister, then we use pattern matching on the snapshot to determine if it is Some, in which case we send it into the Processor as a Snapshot message. Otherwise we do nothing ()
And then just run in all the Replay actions as before.



That is it!
All code is available at github.

Known issues

Newtonsoft JSON.NET typeconverter issue

Newtonsoft JSON.NET seems to have some trouble with F# Map collections, especially when using union types as keys
type UserId = UserId of Guid.
Map<UserId, User>
Gives following nasty exception when trying to deserialize it as a key in a Map
Newtonsoft.Json.JsonSerializationException
  HResult=0x80131500
  Message=Could not convert string 'UserId 8fa6d5a6-9500-4aac-9f3e-a5b918b81c46' to dictionary key type 'dreamstatecoding.core.Model+UserId'. Create a TypeConverter to convert from the string to the key type object. Path 'UserStore.Users['UserId 8fa6d5a6-9500-4aac-9f3e-a5b918b81c46']', line 1, position 69.
  Source=Newtonsoft.Json
  StackTrace:
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary(IDictionary dictionary, JsonReader reader, JsonDictionaryContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at dreamstatecoding.core.AppPersister.getSnapshot(FileInfo fi) in C:\tfs\apps\dreamstatecoding.core\AppPersister.fs:line 104
   at dreamstatecoding.core.AppPersister.GetLatestSnapshotAndActions() in C:\tfs\apps\dreamstatecoding.core\AppPersister.fs:line 113
   at dreamstatecoding.core.AppHolder.InitiateFromLastSnapshot() in C:\tfs\apps\dreamstatecoding.core\AppHolder.fs:line 47
   at test.web.Program.Main(String[] args) in C:\tfs\apps\test\test.web\Program.cs:line 14

Inner Exception 1:
JsonSerializationException: Error converting value "UserId 8fa6d5a6-9500-4aac-9f3e-a5b918b81c46" to type 'dreamstatecoding.core.Model+UserId'. Path 'UserStore.Users['UserId 8fa6d5a6-9500-4aac-9f3e-a5b918b81c46']', line 1, position 69.

Inner Exception 2:
ArgumentException: Could not cast or convert from System.String to dreamstatecoding.core.Model+UserId.
This is fixed and a new version is on github.. Thanks to the really nice #fsharp community on twitter for the help!
Code changes to include the Fable.JsonConverter
let converters =
 [ Fable.JsonConverter () :> JsonConverter ] |> List.toArray :> IList<JsonConverter>

let settings =
 JsonSerializerSettings (
  Converters = converters,
  Formatting = Formatting.Indented,
  NullValueHandling = NullValueHandling.Ignore)
And then just using it in serializing and deserializing as follows:
let json = JsonConvert.SerializeObject(state, settings)
let newState = JsonConvert.DeserializeObject<ApplicationState.ApplicationState>(json, settings)

All code provided as-is. This is copied from my own code-base, May need some additional programming to work. Use for whatever you want, how you want! If you find this helpful, please leave a comment, not required but appreciated! :)

Hope this helps someone out there!
Until next time: Work to Live, Don’t Live to Work