Auto Saving SavedVariables
 

Hi all,

We're looking to come up with a robust solution to the problem many of you have had when it comes to SavedVars not saving when the game closes unpredictably. We want to implement an auto saving system but also include options to opt out and request a save manually. By default, this system will automatically save SavedVar data for each addon, one at a time, with a large (minute(s)) time interval between addon saves. For addons that opt out, there will be a function to manually request that their saved vars be placed at the front of the list to be saved on the next opportunity. This manual request will have a larger cooldown (15min-ish).

We’ll disable auto saving by checking an addon config:
## DisabledSavedVariablesAutoSaving: 1

When a save occurs, we will attempt to do the work to save it out. If the save was not triggered manually and the work takes longer than some reasonable amount of time that we’ll determine, we will abort the save and increment numSaveAttempts. If numSaveAttempts goes higher than some reasonable number of attempts (maybe 3?), we’ll consider the addon to be too intensive to auto save and it won’t try again. Manual saves go through regardless of how long they take.

We would like to get your thoughts on this idea. Do you think this will be helpful to you? Does it give you enough control, and if not, what do you think is it missing? Can you see any serious issues this may cause for you?

Thanks!

I really would love to see this!
My main concern is that right now the table keys in the saved variables are serialized in a random order. This means every time a file is saved, it looks vastly different to the previous revision, which in turn makes it impossible to efficiently create incremental backups. My saved variables folder is close to 250MB right now, which could easily lead to a gigabyte of unnecessary backups every hour.
Ideally this auto save (and also the regular save) would store the keys in alphabetical order and check if the resulting file would be different so it skips writing if there is no change at all.

This sounds great! I don't think it's really something that I will use explicitly, but it's a really good QOL improvement, not just for add-on authors, but any players who use add-ons. As far as explicit uses go, I can see my lazy farmer using it, but I don't know if I'd spend the time to update it to use that.

I think it's a good addition but some ppl mentioned it shouldn't save if you are in a fight, especially not if you are in a dungeon or even trial!

Not sure if the saving will produce any delay.
But I support the idea of Sirinsidiator to compare the savedvars in memory with the file somehow (by help of a hash e.g.) before saving it, if nothing has changed.

Are they saved asynchronous? E.g. via memory snapshot. I can imagine some saved-vars could take a bit longer.

Hey Dan, this is great!

Any news on when it would be possible to receive new mails without need to relog or change map?

I think it would be a better idea to make the option so you have to opt in to have it work. Making it run by default can cause performance issues with addons that store large amounts of cache data to their saved variables until the author gets around to turning the option off.

Something like this:

Maybe the author can mark a savedVar as dirty. The authors know best, if a critical change was made.

I agree that opt in is much preferred. I use more than a few addons that are no longer updated/supported, but still work. (Some just work and others I've fixed so they work again.) I'm unsure how autosaving might impact them or my client's performance. Sure, I could edit all the old addons with "## DisabledSavedVariablesAutoSaving: 1", but that wouldn't be necessary if the system was opt in.

Using the example Dan posted, implement "## EnabledSavedVariablesAutoSaving: 1" to opt in instead.

I also like the dirty flag suggestion. What would really be nice is if specific keys could be marked dirty and only those keys saved during an autosave cycle. However, I have no idea if it's possible to write to SavedVars in part or if it must be written whole every single time.

I would also love it if that feature could be turned off globally via the user settings and maybe start turned off in the first iteration, until we see how it impacts the game on a larger scale.

imo, any behavior change like this that could adversely affect performance should be opt-in; Edit: or perhaps just disabled for out of date add ons.

In Dan's example, running the manual save function registers the saved var to be picked up later by this process. What part of that isn't marking it as dirty?



I highly doubt writing only specific keys would even be possible. Lua memory doesn't look like the code you see in the saved var files. It also doesn't have any stock method of turning data into such code. This is where a serializer comes in. A serializer takes data you give it and encodes it into some sort of string that can be read in by a deserializer to reconstruct the original data later. The serializer for saved vars takes the entire table and writes it into Lua code that can be read in by loadfile() as the deserializer.

If an attempt were to be made, the original file would have to be analyzed with a tokenizer to find where the original data for the key/value pair is in the file to replace it. This would take up a lot more processing than just overwriting the entire thing.

There are repeatable actions and non-repeatable events.
repeatable action: Checkbox in the settings: Worse case: User has to do it again. => Can be handled like now.
non-repeatable event: Completing a quest: Worse case: Gone forever. => Should be saved soon.

As another example:


Craft 5 legendary items with the set crafter. Worst case: They make it again and lose 300k because that wasn't saved. A similar issue would exist with Writ Worthy, especially since some people do a large amount of them.


As another request, would it be possible to have the Save Request function return a value letting the add-on know if the save request has been approved?


Also, it would be great if an add-on could call it twice a few minutes apart, but then have an even longer cooldown. My use case here is actually with the set crafter/writ crafter. If a player adds an item to the queue (saved vars) then they will probably make it pretty quickly after. If it's just the one save request, then I'd save it for the actual crafting, but if it's possible to have two it would be nice to also have it save after a large amount of items are added to the queue. If that's too abusable for you, I'm still excited even for just the one save request option.




I'm seeing a lot of people worried about performance issues from this. How much of a concern is that actually? Maybe if an add-on's saved vars table is less than x amount the game can do default of auto saving, but if it has more than x amount of values it defaults to the current behaviour. Or even just make the 'reasonable' amount of time very short. Depending on the reasonable amount of time chosen, I don't see why there should be an issue with performance, in PvP, trials, or not.

I still don't see how this changes anything. You either need to save now and call the save function, and if it has to, it waits for the cooldown to finish. There is no way around this as per the design of it. If you call for it to save something not so important (as if marking the save dirty), it does the exact same thing. Which of these two processes do you see as different? A suggestion I would make, if this is the direction we're going, is to have the autosave and manual saves share the same cooldown. Something like 5 minutes seems reasonable to me.



An alternative I would suggest is to add a directive in the manifest that would both have an addon opt-in for autosave and configure how often the author wants it to occur. Something like ## AutoSaveInterval: 5 would set the interval for every 5 minutes. This has to be an integer greater than zero or the directive will be ignored.

In addition, a manual save function will be made available to all addons, regardless of this setting. For the sake of this example, the cooldown for an addon is 1 minute regardless of if processed by autosave or manually done so. A scheduler system should be made to handle the order in which addons save their data. This will act as a queue that iterates through one step every 5 seconds. When processed, addons configured for autosave will be rescheduled according to the interval set in their manifest. Manually calling the save function will modify the schedule to run at the the current timestamp, or 1 minute after the timestamp of the last save, whichever is greater.

To clarify, if an addon that's registered for autosave calls the manual function, it will still reschedule itself the same as if it was triggered by autosave. This supports more urgent saving and preserves the autosave feature at the same time. Duplicate entries for a single addon is not allowed. If there an entry already exists, it should be modified appropriately if the requested timestamp is earlier and the queue re-sorted. This design also allows the queue to fall behind if the step doesn't cycle through fast enough if there are too many addons using it.

in general i like the idea of auto savings. in my opinion, one time each 15 or 30 is enough. in what cases you want to save it every 5‘?
what addon needs it?
- how many times per day you gonna change settings?
- merchant addon can read the history, not really needed

maybe some log addons or chat savings, but how many people are using this?

why not a button to save it manually? (with a function that can be called via lua code to be able to allow autosavings via RegisterForUpdate)

and after writing this text i believe, we don’t need SV autosaving

Saving any x minutes where x is below 5 won't happen imo as Dan has already stated in his thread above.
Making the opt in would be better imo, as existing addons need to be changed to adopt it.

If this causes the game to stutter every 5, 15, or 30 mins, people are going to complain the same regardless of how long between. I thought 5 mins between saves would be more reasonable than 15. It doesn't matter if I'll have a use for it, but if it's an option that'll fit into more peoples' needs with hardly any impact, why not? Especially with something that'll effect everyone that plays the game in some way. In all honesty, I don't have a "need" for autosave either and I've crashed the game more times than I care to count.



From the sounds of it, Dan's team hasn't fully figured out how autosave is going to work, let alone ironed out an interval between saves. He posted here for suggestions, I made mine.

I doubt it will cause the game to stutter. If it's something that would cause the game to stutter, I don't think it would be implemented.

Missing the point. It's not likely, but it is the worst that could happen and is the reason Dan suggested having a cooldown for saving in the first place.

It's not mentioned here, but on Gitter Chip told us that the saved variables are written in a background thread, so there should not be any stutter or performance issue, unless you use a single core CPU or limit ESO to one core.

Actual writing is a different stage of the process than serialization. In order to do it while the addon system is running, Lua cannot be allowed to execute any code or you'll have the possibility of data corruption. This means it needs to be done with the UI thread or at least lock it meanwhile. Locking the UI thread either way means the UI will be unresponsive for the duration and when that happens, it ties up the rendering engine as well. It's a cascading effect.

Just because something runs on a different thread doesn't mean thread locking doesn't exist, it's a necessity in certain applications.

Hi everyone,

Thank you so much for the feedback! I have another question, then I'll give a bit more info:

Can you provide me with some numbers on how big your addon's saved var files typically are? As many real examples as you can give me. If you know someone else's I'll take that too. I'm trying to get an idea of what's small, what's large, and what's typical.

As for the feedback, I can tell you that performance is a big focus here, and we will be trying a number of different avenues to ensure we aren't causing spikes. You're not the only ones concerned about that, believe me it's on our minds too. We may look into using some of your suggestions for that, as well.

As for auto on vs auto off, assuming performance is under control, a key to this feature is that not every addon developer is going to think or remember to enable this, nor will the many addons that are fully functional but not maintained anymore. We want this to help as many people as possible while not causing issues. We will strike that balance between speed and size to find a way to allow the most number of addons to leverage it. And priority saving will always be available to those whose addon is too beefy to do it automatically but they still want to save.

So I have 64 add-ons with saved variables. 25 have a file size less than 10 kilobytes.
10 have a file size between 10 and 50 KB
5 are between 50 and 100, and another 5 between 100 and 200.
There's just two between 200 and 1000
For the next few, I'm ignoring the MM00-MM15 files.
2 are between 1000 and 2000, and then there's one at 7.1 MB.


If you include MM, the lowest is 3.8 MB, largest is 8.5, and the average is 6.2 MB, with a deviation of 1.4 MB.

I do have a spreadsheet with the names of the addons and sizes here: https://docs.google.com/spreadsheets...it?usp=sharing


For one of my addons, the Set Crafter, I tried to see what some reasonable sizes would be.


With no use of it, it's about 6 KB (0 items). What I'd consider normal use (20 items) was about 50 KB Moderately heavy (80) was about 160. Very heavy use (200 items) gave me a size of 410.

The Writ Crafter uses 32 KB for me, and I don't think that the size will vary a lot.

With regards to saved vars size, are we talking raw file size? Number of lines? Size of tables? All of the above?

My SavedVars folder weight is ~124 MB - yes, I'm addons addict, more than 60 addons overall (if not count all MM clones) :D Not all of them are enabled on all characters tho...

So, ~124 MegaBytes, 78 files, there is some from uninstalled addons, sure, but their weight aren't big.
Most heavy files are at image. This is only 38 files, so also 40 more with size less than 70 KB

36.4 MB

https://i.imgur.com/nEoxpfd.png
The average is 3.1mb per file.
When looking at my SavedVars folder, I feel like the files with more than 1mb are not important to be auto-saved.

The really large files (MM**.lua and ArkadiusTradeToolsSalesData**.lua) contain sales data from the guild history. If saving data of the current play session fails, the data is simply re-obtained upon the next login when scanning the guild's history. So I don't think these are files that would require auto-saving.

The HarvestMap**.lua files contain locations of crafting resources. If these files are large, then the player typically used some external tool to obtain data, in which case it is also not a big issue, if the data is lost.
However, if the files are small, then they typically only contain data collected by the player themself. In that case the player would be rather sad to lose the collected data from the current play session.
Personally, I think I would use the optional save request for HarvestMap to accommodate for these two different cases.

After trimming some old files my saved variables folder has ~200MB where ~150MB are Master Merchant files. Second place is HarvestMap with 8.15MB and third comes QuickEnchanter with 8MB. Then I have 10 more files with a size between 1 and 6MB, 10 between 100kB and 1MB and the rest (59 files) below 100kB with an average size of 13kB (but most of them are <10kB actually).

My addon savedvariables folder:
I'm not using Master Merchant so my savedvars folder is "only" about 60MB in total

HarvestMap with 8.07 mb.
3 other addons with about 7 mb.
2 with 4 MB where 1 only stores set item data and won't be updated often (only manually) -> Would be cool if the savedvars would only save the "changed addon data" and not the "non changed set items database" on autosave!)
Several with 1-2 mb size
Several >250kb and < 1 mb
Several 10kb to 250kb
Lots of 1kb to 10kb

MM is largest with 213M
CraftStore is next largest at 7.7M
Furniture Catalogue and Inventory Insight are about 1.7M each
Everything else is 700k or smaller

Saved vars directory:

57 files

I'm not sure all addons need an autosave. Maybe just being able to program a save within the addon, will save a lot of unnecessary saves. In Cashier for example, it only needs to be saved when a transaction has occurred with the bank. Otherwise it does not care. I would prefer to not have it autosaving every X minutes wasting processing power when it does not do anything. But if I disable the autosave, I would like to be able to program into the code that it causes a save when its data is changed. So no lost data will occur.

if its true, why when MM saving data after a scan of a new guild i just joined, game become a slide show for 30s at 8-core cpu?

sirinsidiator was talking about the future auto saving of savedvariables, not about anything that currently exists in the game.
btw the MM lag has nothing to do with saving data. the lag does not come from saving something to the hard drive. in the current game version, saving is only done when you logout or /reloadui.
the lag comes from mm scanning and processing the guild sales history.

Avg: ~ 218kb

Holy cow, my request got through?!?!?! I will happily do anything required to help test, integrate, and even design this feature!!!

And, your requested file size information:

Here you go:

I'm the developer for Notebook 2018. A fan of the addon (yes, a fan! lol) directed my attention to this post.

At this point, ANY solution is welcome. Opt-in only for "large" savedvars files strongly preferable. My addons savedvars are generally "small", but the content is only user hand-made, so losses are terrible.

Also, when it is added, I wouldn't mind getting some notification with clear instructions / explanation. Be it a message here or an email as an addon developer. :) It'll be a celebration!