New Guild History API

[sirinsidiator]

Thanks to the tireless efforts of Dan, we are finally getting a new and improved Guild History API with Update 41.
Here's a first look at the new API and how it will work.

* ClearGuildHistoryCache(*integer* _guildId_, *[GuildHistoryEventCategory|#GuildHistoryEventCategory]:nilable* _category_)
** _Returns:_ *bool* _success_

* GetNumGuildHistoryEvents(*integer* _guildId_, *[GuildHistoryEventCategory|#GuildHistoryEventCategory]* _category_)
** _Returns:_ *integer* _numEvents_

* GetGuildHistoryEventIndicesForTimeRange(*integer* _guildId_, *[GuildHistoryEventCategory|#GuildHistoryEventCategory]* _category_, *integer53* _newestTimeS_, *integer53* _oldestTimeS_)
** _Returns:_ *luaindex:nilable* _newestEventIndex_, *luaindex:nilable* _oldestEventIndex_

* GetOldestGuildHistoryEventIndexForUpToDateEventsWithoutGaps(*integer* _guildId_, *[GuildHistoryEventCategory|#GuildHistoryEventCategory]* _category_)
** _Returns:_ *luaindex:nilable* _oldestEventIndex_

* GetGuildHistoryEventId(*integer* _guildId_, *[GuildHistoryEventCategory|#GuildHistoryEventCategory]* _category_, *luaindex* _eventIndex_)
** _Returns:_ *integer53* _eventId_

* GetGuildHistoryRosterEventInfo(*integer* _guildId_, *luaindex* _eventIndex_)
** _Returns:_ *integer53* _eventId_, *integer53* _timestampS_, *[GuildHistoryRosterEvent|#GuildHistoryRosterEvent]* _eventType_, *string* _actingDisplayName_, *string* _targetDisplayName_, *integer:nilable* _rankId_

* GetGuildHistoryBankedItemEventInfo(*integer* _guildId_, *luaindex* _eventIndex_)
** _Returns:_ *integer53* _eventId_, *integer53* _timestampS_, *[GuildHistoryBankedItemEvent|#GuildHistoryBankedItemEvent]* _eventType_, *string* _displayName_, *string* _itemLink_, *integer* _quantity_

* GetGuildHistoryBankedCurrencyEventInfo(*integer* _guildId_, *luaindex* _eventIndex_)
** _Returns:_ *integer53* _eventId_, *integer53* _timestampS_, *[GuildHistoryBankedCurrencyEvent|#GuildHistoryBankedCurrencyEvent]* _eventType_, *string* _displayName_, *[CurrencyType|#CurrencyType]* _currencyType_, *integer* _amount_, *string* _kioskName_

* GetGuildHistoryTraderEventInfo(*integer* _guildId_, *luaindex* _eventIndex_)
** _Returns:_ *integer53* _eventId_, *integer53* _timestampS_, *[GuildHistoryTraderEvent|#GuildHistoryTraderEvent]* _eventType_, *string* _sellerDisplayName_, *string* _buyerDisplayName_, *string* _itemLink_, *integer* _quantity_, *integer* _price_, *integer* _tax_

* GetGuildHistoryMilestoneEventInfo(*integer* _guildId_, *luaindex* _eventIndex_)
** _Returns:_ *integer53* _eventId_, *integer53* _timestampS_, *[GuildHistoryMilestoneEvent|#GuildHistoryMilestoneEvent]* _eventType_

* GetGuildHistoryActivityEventInfo(*integer* _guildId_, *luaindex* _eventIndex_)
** _Returns:_ *integer53* _eventId_, *integer53* _timestampS_, *[GuildHistoryActivityEvent|#GuildHistoryActivityEvent]* _eventType_, *string* _displayName_

* GetGuildHistoryAvAActivityEventInfo(*integer* _guildId_, *luaindex* _eventIndex_)
** _Returns:_ *integer53* _eventId_, *integer53* _timestampS_, *[GuildHistoryAvAActivityEvent|#GuildHistoryAvAActivityEvent]* _eventType_, *string* _displayName_, *integer* _keepId_, *integer* _campaignId_

* CreateGuildHistoryRequest(*integer* _guildId_, *[GuildHistoryEventCategory|#GuildHistoryEventCategory]* _category_, *integer53* _newestTimeS_, *integer53* _oldestTimeS_)
** _Returns:_ *integer* _requestId_

* GetGuildHistoryRequestFlags(*integer* _requestId_)
** _Returns:_ *[GuildHistoryRequestFlags|#GuildHistoryRequestFlags]* _flags_

* IsGuildHistoryRequestComplete(*integer* _requestId_)
** _Returns:_ *bool* _isComplete_

* DoesGuildHistoryHaveOutstandingRequest()
** _Returns:_ *bool* _hasOutstandingRequest_

* RequestMoreGuildHistoryEvents(*integer* _requestId_, *bool* _queueRequestIfOnCooldown_)
** _Returns:_ *[GuildHistoryDataReadyState|#GuildHistoryDataReadyState]* _readyState_

* GetNumGuildHistoryEventRanges(*integer* _guildId_, *[GuildHistoryEventCategory|#GuildHistoryEventCategory]* _category_)
** _Returns:_ *integer* _numGuildHistoryEventRanges_

* GetGuildHistoryEventRangeInfo(*integer* _guildId_, *[GuildHistoryEventCategory|#GuildHistoryEventCategory]* _category_, *luaindex* _rangeIndex_)
** _Returns:_ *integer53* _newestTimeS_, *integer53* _oldestTimeS_, *integer53* _newestEventId_, *integer53* _oldestEventId_

* DestroyGuildHistoryRequest(*integer* _requestId_)
** _Returns:_ *bool* _successfullyDestroyed_

* EVENT_GUILD_HISTORY_CATEGORY_UPDATED (*integer* _guildId_, *[GuildHistoryEventCategory|#GuildHistoryEventCategory]* _eventCategory_, *[GuildHistoryCategoryUpdateFlags|#GuildHistoryCategoryUpdateFlags]* _flags_)

h5. GuildHistoryActivityEvent
* GUILD_HISTORY_ACTIVITY_EVENT_ABOUT_US_EDITED
* GUILD_HISTORY_ACTIVITY_EVENT_MOTD_EDITED
* GUILD_HISTORY_ACTIVITY_EVENT_RECRUITMENT_LISTED
* GUILD_HISTORY_ACTIVITY_EVENT_RECRUITMENT_UNLISTED


h5. GuildHistoryAvAActivityEvent
* GUILD_HISTORY_AVA_ACTIVITY_EVENT_KEEP_CLAIMED
* GUILD_HISTORY_AVA_ACTIVITY_EVENT_KEEP_LOST
* GUILD_HISTORY_AVA_ACTIVITY_EVENT_KEEP_RELEASED


h5. GuildHistoryBankedCurrencyEvent
* GUILD_HISTORY_BANKED_CURRENCY_EVENT_DEPOSITED
* GUILD_HISTORY_BANKED_CURRENCY_EVENT_HERALDRY_EDITED
* GUILD_HISTORY_BANKED_CURRENCY_EVENT_KIOSK_BID
* GUILD_HISTORY_BANKED_CURRENCY_EVENT_KIOSK_BID_REFUND
* GUILD_HISTORY_BANKED_CURRENCY_EVENT_KIOSK_PURCHASED
* GUILD_HISTORY_BANKED_CURRENCY_EVENT_WITHDRAWN


h5. GuildHistoryBankedItemEvent
* GUILD_HISTORY_BANKED_ITEM_EVENT_ADDED
* GUILD_HISTORY_BANKED_ITEM_EVENT_REMOVED


h5. GuildHistoryCategoryUpdateFlags
* GUILD_HISTORY_CATEGORY_UPDATE_FLAG_NEW_INFO
* GUILD_HISTORY_CATEGORY_UPDATE_FLAG_REFRESHED
* GUILD_HISTORY_CATEGORY_UPDATE_FLAG_RESPONSE_RECEIVED


h5. GuildHistoryDataReadyState
* GUILD_HISTORY_DATA_READY_STATE_INVALID_REQUEST
* GUILD_HISTORY_DATA_READY_STATE_ON_COOLDOWN
* GUILD_HISTORY_DATA_READY_STATE_READY
* GUILD_HISTORY_DATA_READY_STATE_RESPONSE_PENDING


h5. GuildHistoryEventCategory
* GUILD_HISTORY_EVENT_CATEGORY_ACTIVITY
* GUILD_HISTORY_EVENT_CATEGORY_AVA_ACTIVITY
* GUILD_HISTORY_EVENT_CATEGORY_BANKED_CURRENCY
* GUILD_HISTORY_EVENT_CATEGORY_BANKED_ITEM
* GUILD_HISTORY_EVENT_CATEGORY_MILESTONE
* GUILD_HISTORY_EVENT_CATEGORY_ROSTER
* GUILD_HISTORY_EVENT_CATEGORY_TRADER


h5. GuildHistoryEventSubcategory
* GUILD_HISTORY_EVENT_SUBCATEGORY_ALL
* GUILD_HISTORY_EVENT_SUBCATEGORY_DEPOSITS
* GUILD_HISTORY_EVENT_SUBCATEGORY_HIRED_TRADER
* GUILD_HISTORY_EVENT_SUBCATEGORY_OWNERSHIP
* GUILD_HISTORY_EVENT_SUBCATEGORY_PURCHASES
* GUILD_HISTORY_EVENT_SUBCATEGORY_WITHDRAWALS


h5. GuildHistoryMilestoneEvent
* GUILD_HISTORY_MILESTONE_EVENT_BANK_LOCKED
* GUILD_HISTORY_MILESTONE_EVENT_BANK_UNLOCKED
* GUILD_HISTORY_MILESTONE_EVENT_KIOSK_LOCKED
* GUILD_HISTORY_MILESTONE_EVENT_KIOSK_UNLOCKED
* GUILD_HISTORY_MILESTONE_EVENT_STORE_LOCKED
* GUILD_HISTORY_MILESTONE_EVENT_STORE_UNLOCKED
* GUILD_HISTORY_MILESTONE_EVENT_TABARD_LOCKED
* GUILD_HISTORY_MILESTONE_EVENT_TABARD_UNLOCKED


h5. GuildHistoryRequestFlags
* GUILD_HISTORY_REQUEST_FLAG_CHUNKS_DIRTY
* GUILD_HISTORY_REQUEST_FLAG_COMPLETE
* GUILD_HISTORY_REQUEST_FLAG_CREATED_FROM_ADDON
* GUILD_HISTORY_REQUEST_FLAG_QUEUED
* GUILD_HISTORY_REQUEST_FLAG_QUEUED_FROM_ADDON
* GUILD_HISTORY_REQUEST_FLAG_RESPONSE_PENDING


h5. GuildHistoryRosterEvent
* GUILD_HISTORY_ROSTER_EVENT_ADDED_TO_BLACKLIST
* GUILD_HISTORY_ROSTER_EVENT_APPLICATION_ACCEPTED
* GUILD_HISTORY_ROSTER_EVENT_APPLICATION_DECLINED
* GUILD_HISTORY_ROSTER_EVENT_DEMOTE
* GUILD_HISTORY_ROSTER_EVENT_EDIT_BLACKLIST_NOTE
* GUILD_HISTORY_ROSTER_EVENT_INVITE
* GUILD_HISTORY_ROSTER_EVENT_JOIN
* GUILD_HISTORY_ROSTER_EVENT_KICKED
* GUILD_HISTORY_ROSTER_EVENT_LEAVE
* GUILD_HISTORY_ROSTER_EVENT_PROMOTE
* GUILD_HISTORY_ROSTER_EVENT_REMOVED_FROM_BLACKLIST


h5. GuildHistoryTraderEvent
* GUILD_HISTORY_TRADER_EVENT_ITEM_SOLD

As you can see in the provided documentation, addons now can request data for specific time ranges without having to load all the history up to that point.
The server will then return the requested data in chunks of 500 entries at a time (up from 100 in the old system), which can be accessed using the new GetNumGuildHistoryEventRanges and GetGuildHistoryEventRangeInfo functions.

Data will be available on the server for 30 days for all categories except the roster and milestone category, which will allow to access data for up to 180 days.
The best part about it is that all the requested data is now cached locally, so that it can be accessed without having to query the server again.
By default the cache will keep data for 30 days, but a setting will be available to choose an arbitrary time frame.

In addition there will also be a helper class that can be used to simplify the process of requesting data from the server.

ZO_GuildHistoryRequest = ZO_InitializingObject:Subclass()
 
function ZO_GuildHistoryRequest:Initialize(guildId, eventCategory, newestTimeS, oldestTimeS)
    self.guildId = guildId
    self.eventCategory = eventCategory
    self.newestTimeS = newestTimeS
    self.oldestTimeS = oldestTimeS
    self.requestId = CreateGuildHistoryRequest(guildId, eventCategory, newestTimeS, oldestTimeS)
end
 
function ZO_GuildHistoryRequest:GetRequestId()
    return self.requestId
end
 
function ZO_GuildHistoryRequest:IsValid()
    return self.requestId ~= 0
end
 
function ZO_GuildHistoryRequest:GetFlags()
    return GetGuildHistoryRequestFlags(self.requestId)
end
 
function ZO_GuildHistoryRequest:IsComplete()
    return IsGuildHistoryRequestComplete(self.requestId)
end
 
function ZO_GuildHistoryRequest:IsRequestQueued()
    return ZO_FlagHelpers.MaskHasFlag(self:GetFlags(), GUILD_HISTORY_REQUEST_FLAG_QUEUED)
end
 
function ZO_GuildHistoryRequest:RequestMoreEvents(queueRequestIfOnCooldown)
    local guildHistoryDataReadyState = RequestMoreGuildHistoryEvents(self.requestId, queueRequestIfOnCooldown)
    return guildHistoryDataReadyState
end

The new system is a complete rewrite, so any addons that directly rely on the old API will need to be updated to work with the new one.

I will also try my best to adapt LibHistoire to the new system so that addons depending on it will continue to work without any or only with minimal changes on their part.
However I decided to remove all the caching functionality from the library, so if you feel like you want to keep any of the old data around, you will need to export it before Update 41 arrives early next year.

Also keep in mind that this is a very early preview and there is still some time for details to change.
If you have questions, suggestions or any other feedback, please feel free to reply to this thread.

[Gamer1986PAN]

Sounds great but i hope someone can update Shissus Guild Tools for that as well since i see a lot of problems coming with that change but its still a very useful addon for all guild work.

[Sharlikran]

Sorry for being slightly off topic, if this is but since it was mentioned...

Shissus Guild Tools was developed with the idea in mind of not having Libraries. It is way to integrated to simply add a few lines here and there and the proper listeners. The code is well written by Shissu but the code is not maintainable in a way that would facilitate the update. So expect Shissus Guild Tools to break in creative and permanent ways.

[JN Slevin]

I was wondering if it would be possible to give us a new permission for guild ranks which lets people see only events associated with them?

Currently virtually all trade guilds use the guild bank gold deposit for donations toward the trader, which some even have a requirement for. This system works rather well, the problem with it is that there is no way for non admin members to see what they donated when without giving them the option to see the guild bank balance and deposits from every other member, which also puts us players in a position where we cannot write an addon for that since other addons could still access this clandestine (for lack of a better word) information. For sales / general / AVA events this information does not really matter that much, but for deposits / withdrawals it does.

To circumvent this dilemma many guilds manually update the note for each member to show how much they donated when, which is very tedious work and can get very quickly out of hand. But this is the only way to show each member when they donated last.

It should also be avoided to make one permission for every event regardless of category since sale events are used by very popular addons which help members make an informed decision about the price of items.

Personally I would propose either making one permission which only affects the bank or making permissions for each category.

My proposal would give members the option to see everything they did in the guild and we could write addons which would help make that information more digestible to members who are not as technical as we are. Furthermore these addons could also reliably show a reminder.

Overall this API change is one of the best news i have heard in a long time. I am really looking forward to testing it out!

[Sharlikran]

To circumvent this dilemma many guilds manually update the note for each member to show how much they donated when, which is very tedious work and can get very quickly out of hand. But this is the only way to show each member when they donated last.

You can make mods that show the player their individual deposits as I have done with Advanced Member Tooltip. The ability to create such mods does not need to specifically use the guild history to do so.

My current obstacle is using LibGuildRoster to add or remove a column on the guild roster without reloading the UI. It appears it can be done but the code for that doesn't seem user/mod author friendly at this point. Unless that has been updated. However, it's in the AMT tooltip anyway.

[JN Slevin]

Originally Posted by Sharlikran You can make mods that show the player their individual deposits as I have done with Advanced Member Tooltip. The ability to create such mods does not need to specifically use the guild history to do so. My current obstacle is using LibGuildRoster to add or remove a column on the guild roster without reloading the UI. It appears it can be done but the code for that doesn't seem user/mod author friendly at this point. Unless that has been updated. However, it's in the AMT tooltip anyway.

This is not possible without giving everyone access to the view bank gold permission. Unless you use the deposit event, that would be possible i agree. But my proposal it would be accessible even without addons which would be helpful for members which do not use addons.

[nightstrike2]

Originally Posted by sirinsidiator The best part about it is that all the requested data is now cached locally, so that it can be accessed without having to query the server again.

Does this mean it will be stored in a saved variable maintained by ZOS somewhere?

By default the cache will keep data for 30 days, but a setting will be available to choose an arbitrary time frame.

Up to what maximum? For instance, I find it extremely useful to look at a full 365 day history to compare sales of event specific items and observe trends throughout the year.

However I decided to remove all the caching functionality from the library, so if you feel like you want to keep any of the old data around, you will need to export it before Update 41 arrives early next year.

Can you provide a way to convert the libhistorie cache into the new ZOS cache?

Also keep in mind that this is a very early preview and there is still some time for details to change. If you have questions, suggestions or any other feedback, please feel free to reply to this thread.

How will the new API handle offline history retention for guilds you are no longer in?

[sirinsidiator]

Originally Posted by nightstrike2 Does this mean it will be stored in a saved variable maintained by ZOS somewhere?

Yes. It won't be a traditional saved variable though. It's in a binary file which will live somewhere in the live folder. There should also be an export function in the game to export it as a text file, but I'm not sure what the state of that is and if it will be available with U41 or get added later.

Originally Posted by nightstrike2 Up to what maximum? For instance, I find it extremely useful to look at a full 365 day history to compare sales of event specific items and observe trends throughout the year.

Whatever you choose. You can set it to a 100 years if you so wish.

Originally Posted by nightstrike2 Can you provide a way to convert the libhistorie cache into the new ZOS cache?

In theory it could be possible with a ridiculous amount of work, but practically no. Which is why I suggest that people export the data in which ever format they need before U41 arrives.

Originally Posted by nightstrike2 How will the new API handle offline history retention for guilds you are no longer in?

I don't know. That's something Dan will have to answer.

[Sharlikran]

I was asking for clarification on what siri meant in a previous conversation when using the word export. As a reminder this doesn't mean it will be in any other format other then Lua and only in the format the mod uses.

In regards to MM this means you need to have linked your sales each day. As long as your sales are up to date then that is your "export" of data. Make sure you have a backup of your GSxxData.lua files from saved variables.

I will be updating my MM documentation and making a seperate post.

[sirinsidiator]

@nightstrike2: I just got info that there will be a setting to allow keeping data for guilds you are no longer a member of. Per default it will be off, since most players won't need it and it will negatively affect loading times.

@JN Slevin: The idea to show members their own gold deposits independently of the bank gold permission seems to be well liked and they are looking into it (no promise it will make it into update 41 though).

[robert.labrie]

How is this going to impact GMs tracking weekly sales for the guild - in particular between the Monday when U41 drops and the Tuesday when the guild traders roll over?

[Baertram]

afai read above you can request the data backwards, just need to do that after the u41 dropped.
So you just need to get all data again to see all guild sales etc. ?

[robert.labrie]

Originally Posted by Baertram afai read above you can request the data backwards, just need to do that after the u41 dropped. So you just need to get all data again to see all guild sales etc. ?

Makes sense. I guess what I'm wondering (hoping) is that MM uses Histoire and that after U41 goes live and I login Histoire will start to build that back history of 30 days of data so that when we roll over on tues and I pop MM I see the whole week.

Older history is lost oh well this is such a huge improvement to performance that it's totally worth it

[Sharlikran]

Only the LibHistorie cache information prior to U41 is deleted, not anything from the existing MM sales data in the LibGuildStore data files. So if you choose last week or the previous week the sales totals will still be there.

[sirinsidiator]

Originally Posted by robert.labrie How is this going to impact GMs tracking weekly sales for the guild - in particular between the Monday when U41 drops and the Tuesday when the guild traders roll over?

The new backend is already collecting data for the past couple months, so when U41 launches there will be access to the full 30 days of trading data from the server side.

[Sharlikran]

Originally Posted by sirinsidiator The new backend is already collecting data for the past couple months, so when U41 launches there will be access to the full 30 days of trading data from the server side.

I didn't see this mentioned before. That's good to know.

[Sharlikran]

I had some time I wasn't playing and once I am able to request the events if my settings are set to 90 days but only 60 are available, will I be able to use GetGuildHistoryEventIndicesForTimeRange() with arbitrary timestamps and simply obtain whatever is in the cache? Will it error if the timestamp is older than the oldest stored event?

[sirinsidiator]

GetGuildHistoryEventIndicesForTimeRange only operates on the cached data (no server interaction involved).
It will simply return nil if the range does not contain any events or give you whatever the cache contains when you call, even if it's incomplete.

In case you do a server request and try to get a range that does not have any events stored on the server, it will finish the request and simply not return data. Doesn't matter if it's past the time limit, or inside the range, but in a guild that has not generate any events in that period.

There are never gonna be any errors for calling any of the new api functions (except for type errors maybe).

Two more things that may also be good to know:

Unlike the old api, the new history will always sort the data when anything is received, so all indices are ordered by time. That means you should always invalidate any indices when the update event fires, since the event it points to may have changed.

The other thing is that the local part of the api is equivalent to the cache. That means GetNumGuildHistoryEvents will return the overall number of events in the cache for that category. There is no extra loading step involved, which also means that longer cache times will mean longer loading times (as before with LibHistoire).

[sirinsidiator]

Today I finally finished rewriting LibHistoire. It took way more time and effort than expected and there are some things I had to cut from v2.0, but it should work for most parts. I do expect some bugs, but will try to fix them over the next couple days.

The new version is compatible with any addon that relied on the old version with minimal or no changes. However, the eventIds in the new api started over from zero and due to differences in how events are stored on the server now, there is sadly no way to map old id64 eventIds to new id53 eventIds.

In order to ensure addon listeners who stored the last processed eventId and use it to limit what the library returns will still receive events, I opted to convert the new eventIds to id64 by adding 3000000000 first. The last ids in the old system on the NA server are somewhere around 2200000000 and for EU around 2700000000, so this way there should not be any collision and it will be easy to convert them back to an id53 later.

Unfortunately this also means that listeners may receive events that have already been processed under the old history api with a different eventId and may end up showing duplicate or incorrect data for a while.

Another change is that the HISTORY_RESCAN_STARTED and HISTORY_RESCAN_ENDED callbacks are now deprecated and won't fire any more. The rescan was originally only required to offer a way to get events that were previously missing from the data returned by the server (due to bugs in the history api) into the library after linking had finished. Now that the cache is part of the game itself, this is no longer possible or needed and if any events end up missing from the server response, ZOS is determined to fix this on their end.

Force linking has also been removed as a feature, since the new api will limit how long data is stored locally anyway and instead the library now offers users a way to make it forget the previously linked range and select the cache segment that is shown in the ingame UI instead. To that end there are two new callbacks LINKED_RANGE_LOST and LINKED_RANGE_FOUND. They will also fire in some other situations like when the cache or the library save data is deleted.

Last but not least, it seems that there is currently a bug with some item links in the new history api, which causes them to miss critical information. This affects writs, crafted potions and any other items that use custom data. I'm not a 100% sure if this is only happening on the PTS or if it will be fixed for live, but fingers crossed.

[Baertram]

Just in case someone needs this:

Finding your local guild history data files on your disk
https://www.esoui.com/forums/showpos...9&postcount=12