I've moved the function to the export part now so the MMPrice will update during export, this should keep the prices up to date and mean that new items get MM data rather than it just being left blank.

It's uploaded in the latest version.

The MM data comes from a look up on each item against MM, so it should happen during the event retrieval part, I can see I have random blanks. As its done when the event happens its possible MM doesnt hold any data for them at the time.
The only thing I could do is move that function to the export part I suppose as it would get out of date by x amount of days otherwise. Not sure how much work that might be, maybe not a lot as I would just need to get data for that field again, and it should then ensure its mostly up to date during export.

I'll have to play with it a little to ensure it doesnt increase the export time by any huge amount.

This should all hopefully be resolved in the latest version I have just released.
Let me know how it goes.

Yeah unfortunately as I have no "in progress" message, you do just have to wait for it to collect all the data on that initial run. Most of the time I change the guild to on and then do a reloadui and wait for it, you can hit the button but its the same process anyway.

I might need to test the gold only part a little more, but all it should to is only write the Gold events to the lua rather than everything.

Your event numbers should match the number of events stored in libhistoire, but it can take a while to get through them all. Which is why I've wanted to work on a processing message or something to give people an idea for that first run.

Thanks for the update, The more feedback the better so I know what is going on, I have tried to test it as much as I can but I appreciate my data is not the same as everyone else's.

I'm going to see what I can get through tonight. Also if you would like to Beta test any of my cutting edge bits of script let me know and I can provide them as I hit milestones.

I'm surprised it caused issues outside the game, I've only experienced in game lag, so that's interesting.
The most recent thing I added was the duplicate function, as it wasn't working properly and for anyone wanting to refresh all the data it meant that it would create multiple copies of the same data.
Of course what that now means its doing is per new event, checking every one of those 55k events, in doing so switching them out from the csv friendly format back into the original format.
As you can imagine that's a huge amount of processing, which when forced it seems to do as quick as it can, by eating all the processing power it can!

If you're happy leaving it disabled for now I will be fixing it this week, maybe today if I can work through the code required quick enough.

What I want to move to, and let me know if you think its a good idea seeing as you probably have more data than most people.
Having the raw data stored in its original format in one .lua which would then only be used to run in the background, do the duplicate checks etc. keep a full up to date copy, and then a second addon used for data management/export which has its own saved variable file.
So thats where the "export" data would go, so I'm thinking some interface to set the time you want, then hit a button and it grabs the data that resides in that time frame, converts it to the csv compatible string and dumps it into the .lua which you can then run your .lua to .csv script against giving you the output, but a much smaller size, with a timeframe variable.

Now the "export" would probably cause some lag, but it would be on-demand, so you can account for it and you'd have a complete message, or a progress message or something, so that you knew what was happening. This export function would then also be where the Gold Only option would reside, I could also move the Guild selection to this too but that might create a lot of extra data that no one wants logging, it might be a bit of trial and error really.

It's a lot of extra stuff that I'll need to write but if it keeps this viable for people with large amounts of data then I can certainly see the benefit, it would also solidify this as a useful upgrade for everyone who uses it.

Now what I would ask, is if you'd like to help develop this into something that does all of these things, so beta test for me essentially?
If so what I can do is pass you my in dev copies of stuff when I get new bits of things to try and you can see how it goes. That way we should see this processing problem before I release it to the wild. As I've said, for my smaller amount of data it seems to run okay, letting LibHistoire control the data flow of new events, but clearly this isnt the case for someone who is seeing many more events than I am.

I'm not sure what machines you are running or whether this is a first time run or what exactly as I've pushed through a lot of tests and my machine copes just fine with just about everything I've thrown at it.
Other than when I was testing some messages that caused the game to time out.
But I only have two guilds with maybe 20k events between them and yes if I hit the refresh it will drop the fps for some amount of time, can't remember how long, but I wasn't bothered as it was the only thing I was testing and wasn't trying to play.

The duplicate function is just so that if you hit the button to refresh the history or for some reason the same events are being recorded again, it will only record it once. It just uses that id field to check, the problem is currently all the data is stored in this csv format so I have to split it back up to do the duplicate check.

The only way I can currently see around that is to use a datastore addon that stores the data in the same format it comes in and then you would have to do an export to get it out in csv friendly format. It's a lot of recoding to do that so I was going to work on that over this week, although now I'm working again the time i have is reduced slightly.

What I did want to check from either of you was whether you've ever managed to let it finish collecting the entirety of the events you had selected? The first time run for a large amount of events will obviously take its time and whilst the lua script is doing things the in game fps does tend to take a plunge.

Yes I am looking at a way to do a time range, i suspect that when i change this to do an "export" rather than the way it currently is that's something that will be easier to implement, but I'll have to wait until later this week to get on it unfortunately.