Thread Tools Display Modes
10/26/20, 07:20 AM   #1
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,566
[open] Archived Data Storage

While writing LibHistoire I've been thinking how to best store the history data in a way that won't require to delete anything in the future, regardless of how many events are stored. The conclusion I have come to is that it's best to segment it by timeframe and then put it into a compressed form. However with the current way saved variables work, this will only get me so far.

There are currently several issues that make it work less than optimal.
  1. saved vars will escape strings on serialization
    • which means we cannot use a binary format for the data
  2. string are limited to a length of up to 1999 characters
    • due to this we have to split longer strings and store them in a table
  3. table indices on numeric tables are explicitly written out
    • this is a huge waste of space and without it, we could easily save 15% and more on the file size
    • due to the way Lua works, it also causes the data in the table to be loaded into the hashed part of a table instead of the indexed part
    • this means accessing values after reloading will have worse performance than if they were still stored in the indexed part
    • it also means the entries on disk will be scrambled when looking at the file (e.g. index 1 is written in a line after index 2)
  4. saved vars are pretty printed
    • removing whitespaces from the file (like what the autosave does), the file would be another 15% smaller
  5. all data is stored in the same file
    • this could be somewhat worked around by creating additional addons for archiving, but the game still loads the data on login

This is why I'd like to ask for a new set of apis which allows addons to store rarely needed data in a way that will minimize impact on loading time and help save disk space.
The idea is to allow addons to store an arbitrary amount of raw (binary) strings as files inside a zip-archive named after the addon.

In order to enable this feature, an addon can put the following directive into its manifest:

Code:
## EnableSavedVariableArchive: 1
The game would then maintain such an archive in memory and only read from or write to disk when regular saved variables would do the same.
An addon can then request specific files to be loaded or saved via the following new functions:

Code:
*boolean* success = LoadArchivedFile(*string* addOnName, *string* pathInArchive)
*boolean* success = SaveArchivedFile(*string* addOnName, *string* pathInArchive, *string* content)
This process should be asynchronous for obvious reason and if it's already running, the methods would just return false. The result will then be delivered via the following events:

Code:
EVENT_ARCHIVED_FILE_LOAD_FINISHED(*string* addOnName, *string* pathInArchive, *string* content)
EVENT_ARCHIVED_FILE_LOAD_FAILED(*string* addOnName, *string* pathInArchive, *ArchivedFileLoadFailureReason* reason)
EVENT_ARCHIVED_FILE_SAVE_FINISHED(*string* addOnName, *string* pathInArchive)
EVENT_ARCHIVED_FILE_SAVE_FAILED(*string* addOnName, *string* pathInArchive, *ArchivedFileSaveFailureReason* reason)
To avoid someone writing something into the archive that could be opened as an executable file, I'd advice to not allow any dots in "pathInArchive". Other than that there should not be any limitations on what can be stored.

In my specific use case I'd like to be able to archive the history per year and month and create a structure like this inside the archive:
  • 2020 (folder)
    • 10 (file)
    • 11 (file)
    • 12 (file)
  • 2021 (folder)
    • 01 (file)

And of course the archive should just be a regular zip file, so it can be opened or modified outside the game.

Last edited by sirinsidiator : 10/26/20 at 07:23 AM.
  Reply With Quote
10/26/20, 08:02 AM   #2
votan
 
votan's Avatar
AddOn Author - Click to view addons
Join Date: Oct 2014
Posts: 577
Some addons "abuse" SavedVars as "database" files. (Including some of mine )
I wished we have some load-/save on demand. Like loading data of a zone on demand. (And not all at once)
This archive files could be locked with an exclusive lock while eso is running to block other processes to do real-time data-exchange.
  Reply With Quote
10/26/20, 08:22 AM   #3
Baertram
Super Moderator
 
Baertram's Avatar
WoWInterface Super Mod
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 4,912
Wouldn't such zip files, "saved on demand", be readable by external tools so that they could read "live information from the server" again?
Or would the execution of function SaveArchivedFile only happen on logout, reloadui, loading screens as normal SV?
  Reply With Quote
10/26/20, 09:05 AM   #4
votan
 
votan's Avatar
AddOn Author - Click to view addons
Join Date: Oct 2014
Posts: 577
Originally Posted by Baertram View Post
Wouldn't such zip files, "saved on demand", be readable by external tools so that they could read "live information from the server" again?
Or would the execution of function SaveArchivedFile only happen on logout, reloadui, loading screens as normal SV?
That's why I said, that these file could be locked by process (eso.exe)
In this case other tools can not open that file, because it is "in use by another process".
  Reply With Quote

ESOUI » Developer Discussions » Wish List » [open] Archived Data Storage

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off