New Guild History API - Page 2

cyxui · 03/11/24, 07:47 PM

Played around with the new api and it seems like a hot mess. The request created from CreateGuildHistoryRequest seems to persist after reloadUI. Not sure if this will eat up the queue CD or not. if someone reloads UI a lot then tons of requests will get created. Not to mention how slow the requests are.

I started a request for one day worth of data (GetTimesStamp(), GetTimeStamp() - 60 * 60 * 24) for a small guild. It has been more than 30min already and its still running. Amount of history should be < 500. Performance is abysmal.

Sharlikran · 03/11/24, 08:38 PM

What I am noticing is that when mods are all fighting for data it seems to cause the UI to freeze. It's been explained to me as on cooldown but however you want to say it. Nothing works when the mods are enabled. For example if I turn all the mods off, I can go get 150 pages of sales data. Once I reload I will have one page of data and all I see is this.

Every category, for every guild is frozen. Nothing works and if I want to get data manually, by pressing E then I get this.

When I haven't requested anything and even if I wait 5 seconds I can't request data.

So I'm not convinced that mods should use the API directly and they should all be using LibHistorie so that if it wants data it can request it, then transmit that to all the mods that are waiting for the data.

If not users are going to be super confused thinking something is broken.

Sharlikran · 03/11/24, 08:41 PM

Originally Posted by cyxui

Played around with the new api and it seems like a hot mess. The request created from CreateGuildHistoryRequest seems to persist after reloadUI. Not sure if this will eat up the queue CD or not. if someone reloads UI a lot then tons of requests will get created. Not to mention how slow the requests are.

I started a request for one day worth of data (GetTimesStamp(), GetTimeStamp() - 60 * 60 * 24) for a small guild. It has been more than 30min already and its still running. Amount of history should be < 500. Performance is abysmal.

I still think there is something to be said about trying to use proactive measures to only request what you need. Tracking how long the person has been offline, checking if the sales exist, and then process the data. If someone is using MM they will already have data so TTC doesn't need to request it. It will already exist in the binary cache.

Regardless of any UI issues currently, when the data is in the cache, it stays there and can be obtained. This isn't like in the past where once you logged out then you had to request 10 days of data again. Once you have the 10 days of data, it's there and it's not going anywhere.

nightstrike2 · 03/11/24, 09:23 PM

Originally Posted by sirinsidiator

Whatever you choose. You can set it to a 100 years if you so wish.

Now that the update has arrived, I can't see any obvious way to set this retention information. Where do you configure it for more than 30 days?

cyxui · 03/11/24, 10:41 PM

Originally Posted by Sharlikran

I still think there is something to be said about trying to use proactive measures to only request what you need. Tracking how long the person has been offline, checking if the sales exist, and then process the data. If someone is using MM they will already have data so TTC doesn't need to request it. It will already exist in the binary cache.

Regardless of any UI issues currently, when the data is in the cache, it stays there and can be obtained. This isn't like in the past where once you logged out then you had to request 10 days of data again. Once you have the 10 days of data, it's there and it's not going anywhere.

In theory thats what SHOULD happen. But it aren't. At least with my experience. Say I requests for 1 day worth of data. 30min later and its done (Seriously? 30min?). Then I log off + login and requests for 1 day worth of data again. Theoretically the game should be smart enough to know that I am only requesting for 5min worth of data since everything between 1 day ago and 5min ago are there. But nope. The game goes on and run the 30min request once more. I would totally expect 5 min worth of data to take minutes not half an hour.

cyxui · 03/11/24, 10:42 PM

Originally Posted by Sharlikran

What I am noticing is that when mods are all fighting for data it seems to cause the UI to freeze. It's been explained to me as on cooldown but however you want to say it. Nothing works when the mods are enabled. For example if I turn all the mods off, I can go get 150 pages of sales data. Once I reload I will have one page of data and all I see is this.

Every category, for every guild is frozen. Nothing works and if I want to get data manually, by pressing E then I get this.

When I haven't requested anything and even if I wait 5 seconds I can't request data.

So I'm not convinced that mods should use the API directly and they should all be using LibHistorie so that if it wants data it can request it, then transmit that to all the mods that are waiting for the data.

If not users are going to be super confused thinking something is broken.

You dont even need multiple addons to break this. Just reload UI twice and boom. The whole thing is locked up.

Sharlikran · 03/12/24, 01:15 AM

Sufice it to say I manipulated the guild history from chat to repair it when it stopped working for me.

1 out of two guilds was repaired, the other I had to clear the cache and then request the data again.

I did this from chat with no mods. So I have simply reported the guild history as broken.

I have a backup of the files so when it breaks they can be compared.

**Baertram** · 03/12/24, 03:34 AM

Originally Posted by cyxui

You dont even need multiple addons to break this. Just reload UI twice and boom. The whole thing is locked up.

As long as I use the Guild History UI to query the past 10 days it's smooth and fast there (all addons off!).
Only if I got LibHistoire enablede solo (which you need to set any guild's category to "Force" then so it updaes that data of the category), or any addon enabled that wants to query the data, then I experience the same as Sharlikran:

I wait forever and cannot get anything except that "Guild history requests can be done once every 2 seconds", even if clicking the normal UI buttons / "Get more" keybind then.
After like 1 hour the banked ites was updated then

For 1 guild...

So I guess every API usage outside of the Guild History UI will lock it at the moment.

sirinsidiator · 03/12/24, 06:08 AM

Originally Posted by nightstrike2

Now that the update has arrived, I can't see any obvious way to set this retention information. Where do you configure it for more than 30 days?

That's one of the things I had to cut from this version of the lib. You can manually adjust it in your UserSettings.txt in the meantime.

peniku8 · 03/12/24, 09:21 AM

Originally Posted by Sharlikran

So I'm not convinced that mods should use the API directly and they should all be using LibHistorie so that if it wants data it can request it, then transmit that to all the mods that are waiting for the data.

Since my addons have been relying on my modded version of Shissu's history scanner, which now broke, I've been thinking about doing this to track member join dates and donations. Is it documented anywhere how I would go about implementing LibHistoire for this kind of purpose? I've talked to Sirinsidiator about this two years ago, when this seemed impossible to me, but I suppose a lot has changed in that time. Is there a function to request chached data for let's say all gank transactions for a guild over a specific time frame?
If this question is too OT I'd be happy to talk about this elsewhere, because I'd love to implement an efficient solution instead of just making another addon that constantly requests data from the server.

Sharlikran · 03/12/24, 11:34 AM

My mod already does that but it relies on the data existing. So say your user joined 2 years ago and the guild history has 180 days, that won't work. A new GM using the mod will not see the join date.

Deposits are also already available for those with permission. Now that users see their own activities then that can be used per user. Although I already show what they deposited using an event.

nightstrike2 · 03/12/24, 12:57 PM

Originally Posted by sirinsidiator

That's one of the things I had to cut from this version of the lib. You can manually adjust it in your UserSettings.txt in the meantime.

Ah, thanks for the pointer. I found the settings in UserSettings.txt. Much appreciated.

Originally Posted by sirinsidiator

Data will be available on the server for 30 days for all categories except the roster and milestone category

It looks like at least for now, the server only has 11 days. Presumably they started with the same 10 as before and are now incrementally adding back up to 30, but they didn't collect 30 days before the U41 drop.

cyxui · 03/12/24, 03:03 PM

Originally Posted by Baertram

As long as I use the Guild History UI to query the past 10 days it's smooth and fast there (all addons off!).
Only if I got LibHistoire enablede solo (which you need to set any guild's category to "Force" then so it updaes that data of the category), or any addon enabled that wants to query the data, then I experience the same as Sharlikran:

I wait forever and cannot get anything except that "Guild history requests can be done once every 2 seconds", even if clicking the normal UI buttons / "Get more" keybind then.
After like 1 hour the banked ites was updated then

For 1 guild...

So I guess every API usage outside of the Guild History UI will lock it at the moment.

Before the update i was getting like 100 per 2min and i thought its slow enough. Now its 30-60min for 500 if you are lucky that the request didnt get stuck.

ImpOfThePerverse · 03/16/24, 05:46 PM

Is there documentation somewhere on how to use the new system? I see function and event signatures, and constants, but nothing giving an overview of basic operations like creating a guild history request and iterating over the results.

I'm used to figuring out game APIs through trial and error, but with requests taking that long to process it doesn't seem like that will be very effective.

My best guess is:

Use CreateGuildHistoryRequest to get the server to start caching events.

Poll with GetGuildHistoryRequestFlags until GUILD_HISTORY_REQUEST_FLAG_COMPLETE is true (unless there's an event I can register for that will let me know?)

Iterate over the results via GetGuildHistoryEventBasicInfo (or one of the event category specific variants of it) in a for loop, using starting and ending event indices obtained via GetGuildHistoryEventIndicesForTimeRange.

Is this close? What happens when there are more than 500 events in the range specified?

Thank you, any help will be greatly appreciated.

Sharlikran · 03/16/24, 06:02 PM

At this point it's best to explain what you want to do or upgrade. The reason is if you use the low level API (meaning what you see in the docs) then it could cause your mod to appear to work properly. However, LibHistorie may not be able to function because you did something that is taboo, and it put everything on cooldown by accident. Like for example, don't try to fix mods like Shissu's.

If you want to make something from scratch, go ahead. Read the the LibHistorie description page.

ImpOfThePerverse · 03/16/24, 07:15 PM

Oh, sorry, I meant writing something to work directly with ESO's updated guild history API. I wasn't planning to make use of LibHistoire.

I would like to request guild event history within a specific category and iterate over the results.

Sharlikran · 03/16/24, 07:53 PM

If it's not for a mod, so you don't create incompatibility or lock up functioning mods, then sorry, never mind. It's a large binary file you don't request anything. I apologize for misunderstanding, sorry for interrupting.

ImpOfThePerverse · 03/16/24, 08:17 PM

The addon won't be in general circulation, or run alongside other addons that create guild history requests. To keep things simple I would prefer to just work directly with the ESO API.

sirinsidiator · 03/17/24, 08:17 AM

You should not use the low-level api directly. Instead it's advised to utilize the high-level api found here.

But you will run into a lot of issues if you do that right now, since the game currently has a couple bugs in regards to requesting data. If you don't want to spend countless hours navigating around them, you are better off relying on LibHistoire.

If you still insist on not using it after hearing that, be warned that you will inadvertently interfere with operation of the library and make it slower. So don't blame me when it ends up taking even longer to request data.

ImpOfThePerverse · 03/17/24, 11:32 AM

Ah, gotcha, thanks. Dealing with a buggy API can be an enormous, onerous headache. Hopefully ZOS cleans it up in the not too distant future. Still, they should probably try to communicate their new system better - I'm sure right now there are a bunch of addon authors banging away at the API trying to figure it out, jamming the servers with messed up history requests.