New Guild History API
 

Thanks to the tireless efforts of Dan, we are finally getting a new and improved Guild History API with Update 41.
Here's a first look at the new API and how it will work.

As you can see in the provided documentation, addons now can request data for specific time ranges without having to load all the history up to that point.
The server will then return the requested data in chunks of 500 entries at a time (up from 100 in the old system), which can be accessed using the new GetNumGuildHistoryEventRanges and GetGuildHistoryEventRangeInfo functions.

Data will be available on the server for 30 days for all categories except the roster and milestone category, which will allow to access data for up to 180 days.
The best part about it is that all the requested data is now cached locally, so that it can be accessed without having to query the server again.
By default the cache will keep data for 30 days, but a setting will be available to choose an arbitrary time frame.

In addition there will also be a helper class that can be used to simplify the process of requesting data from the server.


The new system is a complete rewrite, so any addons that directly rely on the old API will need to be updated to work with the new one.

I will also try my best to adapt LibHistoire to the new system so that addons depending on it will continue to work without any or only with minimal changes on their part.
However I decided to remove all the caching functionality from the library, so if you feel like you want to keep any of the old data around, you will need to export it before Update 41 arrives early next year.

Also keep in mind that this is a very early preview and there is still some time for details to change.
If you have questions, suggestions or any other feedback, please feel free to reply to this thread.

Sounds great but i hope someone can update Shissus Guild Tools for that as well since i see a lot of problems coming with that change but its still a very useful addon for all guild work.

Sorry for being slightly off topic, if this is but since it was mentioned...

Shissus Guild Tools was developed with the idea in mind of not having Libraries. It is way to integrated to simply add a few lines here and there and the proper listeners. The code is well written by Shissu but the code is not maintainable in a way that would facilitate the update. So expect Shissus Guild Tools to break in creative and permanent ways.

Permissions
 

I was wondering if it would be possible to give us a new permission for guild ranks which lets people see only events associated with them?

Currently virtually all trade guilds use the guild bank gold deposit for donations toward the trader, which some even have a requirement for. This system works rather well, the problem with it is that there is no way for non admin members to see what they donated when without giving them the option to see the guild bank balance and deposits from every other member, which also puts us players in a position where we cannot write an addon for that since other addons could still access this clandestine (for lack of a better word) information. For sales / general / AVA events this information does not really matter that much, but for deposits / withdrawals it does.

To circumvent this dilemma many guilds manually update the note for each member to show how much they donated when, which is very tedious work and can get very quickly out of hand. But this is the only way to show each member when they donated last.

It should also be avoided to make one permission for every event regardless of category since sale events are used by very popular addons which help members make an informed decision about the price of items.

Personally I would propose either making one permission which only affects the bank or making permissions for each category.

My proposal would give members the option to see everything they did in the guild and we could write addons which would help make that information more digestible to members who are not as technical as we are. Furthermore these addons could also reliably show a reminder.

Overall this API change is one of the best news i have heard in a long time. I am really looking forward to testing it out!

You can make mods that show the player their individual deposits as I have done with Advanced Member Tooltip. The ability to create such mods does not need to specifically use the guild history to do so.

My current obstacle is using LibGuildRoster to add or remove a column on the guild roster without reloading the UI. It appears it can be done but the code for that doesn't seem user/mod author friendly at this point. Unless that has been updated. However, it's in the AMT tooltip anyway.

This is not possible without giving everyone access to the view bank gold permission. Unless you use the deposit event, that would be possible i agree. But my proposal it would be accessible even without addons which would be helpful for members which do not use addons.

Does this mean it will be stored in a saved variable maintained by ZOS somewhere?

Up to what maximum? For instance, I find it extremely useful to look at a full 365 day history to compare sales of event specific items and observe trends throughout the year.

Can you provide a way to convert the libhistorie cache into the new ZOS cache?

How will the new API handle offline history retention for guilds you are no longer in?

Yes. It won't be a traditional saved variable though. It's in a binary file which will live somewhere in the live folder. There should also be an export function in the game to export it as a text file, but I'm not sure what the state of that is and if it will be available with U41 or get added later.

Whatever you choose. You can set it to a 100 years if you so wish.

In theory it could be possible with a ridiculous amount of work, but practically no. Which is why I suggest that people export the data in which ever format they need before U41 arrives.

I don't know. That's something Dan will have to answer.

I was asking for clarification on what siri meant in a previous conversation when using the word export. As a reminder this doesn't mean it will be in any other format other then Lua and only in the format the mod uses.

In regards to MM this means you need to have linked your sales each day. As long as your sales are up to date then that is your "export" of data. Make sure you have a backup of your GSxxData.lua files from saved variables.

I will be updating my MM documentation and making a seperate post.

@nightstrike2: I just got info that there will be a setting to allow keeping data for guilds you are no longer a member of. Per default it will be off, since most players won't need it and it will negatively affect loading times.

@JN Slevin: The idea to show members their own gold deposits independently of the bank gold permission seems to be well liked and they are looking into it (no promise it will make it into update 41 though).

How is this going to impact GMs tracking weekly sales for the guild - in particular between the Monday when U41 drops and the Tuesday when the guild traders roll over?

afai read above you can request the data backwards, just need to do that after the u41 dropped.
So you just need to get all data again to see all guild sales etc. ?

Makes sense. I guess what I'm wondering (hoping) is that MM uses Histoire and that after U41 goes live and I login Histoire will start to build that back history of 30 days of data so that when we roll over on tues and I pop MM I see the whole week.

Older history is lost oh well this is such a huge improvement to performance that it's totally worth it

Only the LibHistorie cache information prior to U41 is deleted, not anything from the existing MM sales data in the LibGuildStore data files. So if you choose last week or the previous week the sales totals will still be there.

The new backend is already collecting data for the past couple months, so when U41 launches there will be access to the full 30 days of trading data from the server side.

I didn't see this mentioned before. That's good to know.

I had some time I wasn't playing and once I am able to request the events if my settings are set to 90 days but only 60 are available, will I be able to use GetGuildHistoryEventIndicesForTimeRange() with arbitrary timestamps and simply obtain whatever is in the cache? Will it error if the timestamp is older than the oldest stored event?

GetGuildHistoryEventIndicesForTimeRange only operates on the cached data (no server interaction involved).
It will simply return nil if the range does not contain any events or give you whatever the cache contains when you call, even if it's incomplete.

In case you do a server request and try to get a range that does not have any events stored on the server, it will finish the request and simply not return data. Doesn't matter if it's past the time limit, or inside the range, but in a guild that has not generate any events in that period.

There are never gonna be any errors for calling any of the new api functions (except for type errors maybe).

Two more things that may also be good to know:

Unlike the old api, the new history will always sort the data when anything is received, so all indices are ordered by time. That means you should always invalidate any indices when the update event fires, since the event it points to may have changed.

The other thing is that the local part of the api is equivalent to the cache. That means GetNumGuildHistoryEvents will return the overall number of events in the cache for that category. There is no extra loading step involved, which also means that longer cache times will mean longer loading times (as before with LibHistoire).

Today I finally finished rewriting LibHistoire. It took way more time and effort than expected and there are some things I had to cut from v2.0, but it should work for most parts. I do expect some bugs, but will try to fix them over the next couple days.

The new version is compatible with any addon that relied on the old version with minimal or no changes. However, the eventIds in the new api started over from zero and due to differences in how events are stored on the server now, there is sadly no way to map old id64 eventIds to new id53 eventIds.

In order to ensure addon listeners who stored the last processed eventId and use it to limit what the library returns will still receive events, I opted to convert the new eventIds to id64 by adding 3000000000 first. The last ids in the old system on the NA server are somewhere around 2200000000 and for EU around 2700000000, so this way there should not be any collision and it will be easy to convert them back to an id53 later.

Unfortunately this also means that listeners may receive events that have already been processed under the old history api with a different eventId and may end up showing duplicate or incorrect data for a while.

Another change is that the HISTORY_RESCAN_STARTED and HISTORY_RESCAN_ENDED callbacks are now deprecated and won't fire any more. The rescan was originally only required to offer a way to get events that were previously missing from the data returned by the server (due to bugs in the history api) into the library after linking had finished. Now that the cache is part of the game itself, this is no longer possible or needed and if any events end up missing from the server response, ZOS is determined to fix this on their end.

Force linking has also been removed as a feature, since the new api will limit how long data is stored locally anyway and instead the library now offers users a way to make it forget the previously linked range and select the cache segment that is shown in the ingame UI instead. To that end there are two new callbacks LINKED_RANGE_LOST and LINKED_RANGE_FOUND. They will also fire in some other situations like when the cache or the library save data is deleted.

Last but not least, it seems that there is currently a bug with some item links in the new history api, which causes them to miss critical information. This affects writs, crafted potions and any other items that use custom data. I'm not a 100% sure if this is only happening on the PTS or if it will be fixed for live, but fingers crossed.

Just in case someone needs this:

Finding your local guild history data files on your disk
https://www.esoui.com/forums/showpos...9&postcount=12

Played around with the new api and it seems like a hot mess. The request created from CreateGuildHistoryRequest seems to persist after reloadUI. Not sure if this will eat up the queue CD or not. if someone reloads UI a lot then tons of requests will get created. Not to mention how slow the requests are.

I started a request for one day worth of data (GetTimesStamp(), GetTimeStamp() - 60 * 60 * 24) for a small guild. It has been more than 30min already and its still running. Amount of history should be < 500. Performance is abysmal.

What I am noticing is that when mods are all fighting for data it seems to cause the UI to freeze. It's been explained to me as on cooldown but however you want to say it. Nothing works when the mods are enabled. For example if I turn all the mods off, I can go get 150 pages of sales data. Once I reload I will have one page of data and all I see is this.



Every category, for every guild is frozen. Nothing works and if I want to get data manually, by pressing E then I get this.



When I haven't requested anything and even if I wait 5 seconds I can't request data.

So I'm not convinced that mods should use the API directly and they should all be using LibHistorie so that if it wants data it can request it, then transmit that to all the mods that are waiting for the data.

If not users are going to be super confused thinking something is broken.

I still think there is something to be said about trying to use proactive measures to only request what you need. Tracking how long the person has been offline, checking if the sales exist, and then process the data. If someone is using MM they will already have data so TTC doesn't need to request it. It will already exist in the binary cache.

Regardless of any UI issues currently, when the data is in the cache, it stays there and can be obtained. This isn't like in the past where once you logged out then you had to request 10 days of data again. Once you have the 10 days of data, it's there and it's not going anywhere.

Now that the update has arrived, I can't see any obvious way to set this retention information. Where do you configure it for more than 30 days?

In theory thats what SHOULD happen. But it aren't. At least with my experience. Say I requests for 1 day worth of data. 30min later and its done (Seriously? 30min?). Then I log off + login and requests for 1 day worth of data again. Theoretically the game should be smart enough to know that I am only requesting for 5min worth of data since everything between 1 day ago and 5min ago are there. But nope. The game goes on and run the 30min request once more. I would totally expect 5 min worth of data to take minutes not half an hour.

You dont even need multiple addons to break this. Just reload UI twice and boom. The whole thing is locked up.

Sufice it to say I manipulated the guild history from chat to repair it when it stopped working for me.

1 out of two guilds was repaired, the other I had to clear the cache and then request the data again.

I did this from chat with no mods. So I have simply reported the guild history as broken.

I have a backup of the files so when it breaks they can be compared.

As long as I use the Guild History UI to query the past 10 days it's smooth and fast there (all addons off!).
Only if I got LibHistoire enablede solo (which you need to set any guild's category to "Force" then so it updaes that data of the category), or any addon enabled that wants to query the data, then I experience the same as Sharlikran:

I wait forever and cannot get anything except that "Guild history requests can be done once every 2 seconds", even if clicking the normal UI buttons / "Get more" keybind then.
After like 1 hour the banked ites was updated then :) For 1 guild...

So I guess every API usage outside of the Guild History UI will lock it at the moment.

That's one of the things I had to cut from this version of the lib. You can manually adjust it in your UserSettings.txt in the meantime.

Since my addons have been relying on my modded version of Shissu's history scanner, which now broke, I've been thinking about doing this to track member join dates and donations. Is it documented anywhere how I would go about implementing LibHistoire for this kind of purpose? I've talked to Sirinsidiator about this two years ago, when this seemed impossible to me, but I suppose a lot has changed in that time. Is there a function to request chached data for let's say all gank transactions for a guild over a specific time frame?
If this question is too OT I'd be happy to talk about this elsewhere, because I'd love to implement an efficient solution instead of just making another addon that constantly requests data from the server.

My mod already does that but it relies on the data existing. So say your user joined 2 years ago and the guild history has 180 days, that won't work. A new GM using the mod will not see the join date.

Deposits are also already available for those with permission. Now that users see their own activities then that can be used per user. Although I already show what they deposited using an event.

Ah, thanks for the pointer. I found the settings in UserSettings.txt. Much appreciated.

It looks like at least for now, the server only has 11 days. Presumably they started with the same 10 as before and are now incrementally adding back up to 30, but they didn't collect 30 days before the U41 drop.

Before the update i was getting like 100 per 2min and i thought its slow enough. Now its 30-60min for 500 if you are lucky that the request didnt get stuck.

Is there documentation somewhere on how to use the new system? I see function and event signatures, and constants, but nothing giving an overview of basic operations like creating a guild history request and iterating over the results.

I'm used to figuring out game APIs through trial and error, but with requests taking that long to process it doesn't seem like that will be very effective.

My best guess is:

Use CreateGuildHistoryRequest to get the server to start caching events.

Poll with GetGuildHistoryRequestFlags until GUILD_HISTORY_REQUEST_FLAG_COMPLETE is true (unless there's an event I can register for that will let me know?)

Iterate over the results via GetGuildHistoryEventBasicInfo (or one of the event category specific variants of it) in a for loop, using starting and ending event indices obtained via GetGuildHistoryEventIndicesForTimeRange.

Is this close? What happens when there are more than 500 events in the range specified?

Thank you, any help will be greatly appreciated.

At this point it's best to explain what you want to do or upgrade. The reason is if you use the low level API (meaning what you see in the docs) then it could cause your mod to appear to work properly. However, LibHistorie may not be able to function because you did something that is taboo, and it put everything on cooldown by accident. Like for example, don't try to fix mods like Shissu's.

If you want to make something from scratch, go ahead. Read the the LibHistorie description page.

Oh, sorry, I meant writing something to work directly with ESO's updated guild history API. I wasn't planning to make use of LibHistoire.

I would like to request guild event history within a specific category and iterate over the results.

If it's not for a mod, so you don't create incompatibility or lock up functioning mods, then sorry, never mind. It's a large binary file you don't request anything. I apologize for misunderstanding, sorry for interrupting.

The addon won't be in general circulation, or run alongside other addons that create guild history requests. To keep things simple I would prefer to just work directly with the ESO API.

You should not use the low-level api directly. Instead it's advised to utilize the high-level api found here.

But you will run into a lot of issues if you do that right now, since the game currently has a couple bugs in regards to requesting data. If you don't want to spend countless hours navigating around them, you are better off relying on LibHistoire.

If you still insist on not using it after hearing that, be warned that you will inadvertently interfere with operation of the library and make it slower. So don't blame me when it ends up taking even longer to request data. ;)

Ah, gotcha, thanks. Dealing with a buggy API can be an enormous, onerous headache. Hopefully ZOS cleans it up in the not too distant future. Still, they should probably try to communicate their new system better - I'm sure right now there are a bunch of addon authors banging away at the API trying to figure it out, jamming the servers with messed up history requests.

Is the low level API the list of functions on the first page?

Also, to Imp's point, the API that you linked similarly doesn't have an overview, it's just a list of functions (and their implementation, obviously)

Known issues with Update 41

* Mod authors cannot check duplicate sales between old and new server data
* Item links from the server do not contain extended information
* Guild history categories may become unusable

To sum up the known issues, mod authors can not cross reference any previous sales data with the current sales data from the server and look for duplicate sales. Item links don't contain any extended information such as writ vouchers, potion effects, and possibly effects for glyphs. In addition to that you may have all green bars but switch characters and any Trader category could become stuck and you won't be able to use the forward arrow key or E Show More to request more data.

Both correct, as far as I can say.

We do not have any detail docu, thus reading the ZOs code is how we do it (similar in addons then) as it shows how it should work.

UESP has a diff of the current API version versus the last version that lists functions that were added and removed.

https://esodata.uesp.net/current/apidiff.txt

There are quite a few more changes to the guild history API than were listed in ZOS's API patch notes. The first thing I did when trying to debug the addon I'm trying to update was to get VS Code set up for Lua as per this thread:

https://www.esoui.com/forums/showthread.php?t=9875

The function signatures found here are out of date, so I went through and deleted the functions that look guild history related that were listed as removed in the UESP diff. That at least points out which parts of the addon I'm updating are broken.

The full, updated API is linked below, but there's no explanation of the functions or any kind of overview of the guild history API:

https://www.esoui.com/forums/attachm...3&d=1709571451

Thanks for the reminder, updated it now.

Awesome! That will help considerably.


Where I'm getting stuck is in determining when a guild history request has been completed. I have a test addon that generates a request for 1 hour's worth of data when I enter a chat command. I can then check the status of the request or view the results via additional chat commands.

When I check status, DoesGuildHistoryHaveOutstandingRequest will return false, which makes it sound like all guild history requests are complete, but doesn't seem like a good indicator because presumably it will return true if another addon has outstanding requests.

IsGuildHistoryRequestComplete returns false, and GetGuildHistoryRequestFlags returns GUILD_HISTORY_REQUEST_FLAG_CREATED_FROM_ADDON as the only positive flag, which makes it sound like the request is not complete.

If I check the results it lists events as having been returned, which, if results are cached in chunks of 500, should mean the request is complete, unless there are some multiple of 500 results, in which case there could be additional chunks incoming. I'm getting just 1 result currently though, so that should mean the request is complete.

I see there's an event, EVENT_GUILD_HISTORY_CATEGORY_UPDATED, that I could register for. Does it get sent when a new chunk gets delivered in that category? Not sure what the flags it returns mean though so I'm not sure how to use it.


I did see this post over on the elder scrolls online forums:

https://forums.elderscrollsonline.co...ry-time#latest

It makes me wonder what happens when you create a request from some timestamp up until the current timestamp. Do new guild history events take some time register on the server? If so, does the server wait until they've registered before returning guild history request results, or could a guild history request possibly return incomplete results if a request is made before all of the events within its time window have registered?

From ZOS code:
Event gets registered here: https://github.com/esoui/esoui/blob/...0C123-L150C140
And calls self: OnCategoryUpdated(guildId, eventCategory, flags)
https://github.com/esoui/esoui/blob/...nager.lua#L154

It creates categoryData here:
local categoryData = guildData:GetEventCategoryData(eventCategory)

Which should be an object of ZO_GuildHistoryEventCategoryData
Which should then use (via this call: https://github.com/esoui/esoui/blob/...nager.lua#L158) this function here:
ZO_GuildHistoryEventCategoryData:OnCategoryUpdated(flags)

https://github.com/esoui/esoui/blob/..._data.lua#L365
-> ZO_FlagHelpers.MaskHasFlag(flags, GUILD_HISTORY_CATEGORY_UPDATE_FLAG_REFRESHED)
GUILD_HISTORY_CATEGORY_UPDATE_FLAG_NEW_INFO = 2
GUILD_HISTORY_CATEGORY_UPDATE_FLAG_REFRESHED = 4
GUILD_HISTORY_CATEGORY_UPDATE_FLAG_RESPONSE_RECEIVED = 1


Hope this helps with finding what the flags are and do.

For every other question you go there, I geuss somene more experienced with that guild history needs to step in here.

I'm still not 100% sure about how to interpret all of the flags, but the event seems to fire with the NEW_INFO flag when a new guild history event shows up in the guild history on the server. That could be used to trigger a new guild history request if you require up to the minute results in your cache, but I don't think it has anything to do with the requests themselves. I'm also not sure if it fires for each new guild history event, or for batches of events.

I've finally gotten a request to complete. I tried requesting a day's worth of data from 1 week ago, and from 24 hours ago, and both requests completed pretty much immediately (IsGuildHistoryRequestComplete returns true, GUILD_HISTORY_REQUEST_FLAG_COMPLETE is the only positive flag, and there are results in the cache for that time window.)

Requests for more recent data produce results in the cache but the requests are never flagged as complete. My guess is it's a server syncing issue? The server can't guarantee that all of the latest results are in, so the request isn't complete. I haven't narrowed down how recent a request can be and still complete, if that's what's going on, but there probably won't be a reliable number if it's tied to server loads and whatnot.

Whether or not a request is complete or complete-able is not based on server load. If you create a request for a range, it should be marked as complete if one of two things happens:
1. We already have events with no gaps that cover the entire time range of the request
2. We get a response from the server and it is empty, does not specify a next id, or the next id is an id we already have

As for the flags, GUILD_HISTORY_CATEGORY_UPDATE_FLAGs:
RESPONSE_RECEIVED refers to getting a response back from a client request (as opposed to the server pushing a new event)
NEW_INFO either means we received new events we didn't have before, or the category is now considered "up to date" which can happen if we requested latest info and got nothing new back but even knowing that is new info
REFRESHED refers to all the data needs to be reevaluated, right now only happening when permissions change

and GUILD_HISTORY_REQUEST_FLAG:
COMPLETE - We have gotten all events there are to get for the given time range and category
RESPONSE_PENDING - A request message from the client is in flight being processed by the server, waiting for a response
RESPONSE_PENDING_FROM_ADDON - Pairs with RESPONSE_PENDING, denoting that it was an add-on request. Not really used for anything atm tbh.
CHUNKS_DIRTY - Used internally to track updating and saving ranges
QUEUED - Request was made while on cooldown and allowed to queue, so it's in the queue waiting for cooldown to end so it can send to the server. Mutually exclusive with RESPONSE_PENDING
QUEUED_FROM_ADDON - Pairs with QUEUED, used internally to ensure requests from add-ons use add-on cooldown. Will be set if either the request was created by an add-on or if the call to RequestMoreEvents was fired from an add-on.

Note that DoesGuildHistoryHaveOutstandingRequest() specifically returns true if a request had an in flight message, i.e. RESPONSE_PENDING. Queued requests do not count as Outstanding. Requests that aren't complete do not count as Outstanding.

Also note that whenever we talk about gaps and ranges, the reason for the seemingly convoluted setup is because it facilitates being able to request events out of order. We use ranges and gaps to know where the wholes are in our requests so we don't miss events. In the old version of history, you could only request starting from now and going back in time. There was no way to get a specific range of time, which was what made it so annoying for add-ons like MM. Now we have a lot more control, but with that control comes more complication. We're working through those growing pains now. I appreciate your patience while we iron out the kinks. There are a lot of fixes coming thanks to the help of Siri and Sharlikran working with us over the last several weeks.