0 of 0

File information

Last updated

Original upload

Created by

Dlinny_Lag

Uploaded by

DlinnyLag

Virus scan

Safe to use

31 comments

  1. DlinnyLag
    DlinnyLag
    • member
    • 14 kudos
    Locked
    Sticky
    Dictionaries of sets and dictionaries of dictionaries are postponed. Probably, forever, if they will be useless for modders.
    Leave comments with explanation how such dictionaries could be usefull in your mods. 
  2. DeadlyStr1ke
    DeadlyStr1ke
    • member
    • 36 kudos
    Hey there, first and foremost, thank you for this utility!

    I have a little question regarding using it to save the data I have stored, since I'm pretty new to using this and understanding the workflow. What I mean by that is, lets say I have a quest with a script attached to it, that would create the array and store the infromation of it since it's a property. Before I'd go something like this.

    Scriptname DSTest extends Quest

    Form[] Property storageRefs Auto

    Event OnInit()
    storageRefs = new ObjectReference[0]
    Debug.Notification("Master storage created with size of " + storageRefs.length)
    EndEvent

    And later use it by accessing it like so:

    Quest Property MyQuestArrays Auto

    DSTest storageStuff = MyQuestArrays as DSTest 
    storageStuff.storageRefs.Add(self) ; or anything else I'd need to do with it


    Now, I'm curious how would I go about storing the created array and all the info in it in a quest similar to this and later accessing it from outside scripts?

    Thank you for your answer in advance!
    1. DlinnyLag
      DlinnyLag
      • member
      • 14 kudos
      In a sample you've provided, data is referenced by DSTest.storageRefs property and game engine saves the data along with the rest game state.

      F4DS provides different option to refer data and how it is saved.
      1) Data is referenced by a Keyword object. It is the identifier of a globally accesible array/set/dictionary object. Yes, you can work with the same array/set/dictionary object from different scripts at any moment, even simultaneously. Actually identifier of the array/set/dictionary is the combination of 2 things - Keyword and exact type of structure. For example, StringDictString and StringDictInt are different objects even if they are referenced by same Keyword. It could be said that array/set/dictionary object is attached to a Keyword, but I would prefer to avoid such terminology. Array/set/dictionary object is not attached to any game object. It just has an ID part of which is identical to a Keyword. You should think that each F4DS's array/set/dictionary is a global object that is not attached to anything.
      2) Data is saved along with other game data in F4SE co-save file. You can consider that F4DS data is saved to the game session, like all other data saved by game engine to the save files.

      Many examples can be found in LiP Framework and its mod samples.
      Certain example:
      Take a look at ChangesHandler.psc (part of LiP framework), ChangesProcessor.psc (part of LiP Effects mod) and EffectsMain.psc (part of LiP Effects mod). All of them has a function IsInAAFScene that checks content of a dictionary. In all implementations dictionary accessed by Keyword that stored in a variable. I've just ensured that the variable stores a reference to the same Keyword in all implementations. Please note that mentioned scripts just reads data from the dictionary. The dictionary is updated by another script - AAFHandler.psc
    2. DeadlyStr1ke
      DeadlyStr1ke
      • member
      • 36 kudos
      That is just amazing, the way it is saved, handled and the versatility of it when it comes down to accessing it.. this is what I was looking for in my next big project. Thank you for the reply and for this life-saving tool!
    3. DeadlyStr1ke
      DeadlyStr1ke
      • member
      • 36 kudos
      [SOLVED - SOLUTION BELOW]

      Hey there, got one more question. So, I'm storing some object references of spawned misc items in the new Data Structures arrays, to later delete them on trigger. However, if a player picks up any of the item and then the script attempts to mark them for delete (via Disable()+Delete()), I crash. I triple tested it and know that this is the cause, the log is showing the element of the picked up item as:

        FF000E6C-40
        InvalidForm
        FF001067-40

      So, my question is, could I possibly somehow check in the script, kinda like this or something to have it skip the invalid element, remove it and continue (iterator is an int for While iteration):

      ObjectReference refTarget
      If DS:FormArray.Get(MyArray, iterator) as ObjectReference
      Do stuff
      Else
      DS:FormArray.Remove(MyArray, iterator)
      EndIf

      In the source script of DS:FormArray it says that it "return default value if" and I'm a bit lost, since it's not a false and I guess it's not a -1 either? What would I check against to just skip the InvalidForm in the array, just remove it and have the script move on from there.
      Thank you for your reply in advance!

      EDIT: Okay, I pinned it down to specifically the part where it gets the object reference refTarget = DS:FormArray.Get(MyArray, iterator) as ObjectReference

      This produces the crash upon item pickup and iteration to delete the items spawned. If I iterate through the item references, but disable the disable() and delete() but keep the part where it removes the elements from the array in the iteration, I can freely pick up the spawned items cleared from the array, force cleanup without a crash. But should I pick up any of the items spawned before the cleanup - I get a ctd.

      [SOLUTION]
      When working with such things as Misc Item, storing them in an array but having the ability to pick them up into players inventory, it is crucial to work with Dual arrays here. Here's the workflow:

      DS:FormArray.Create(MyArrayCleared)
      DS:FormArray.AppendArray(MyArrayCleared,MyArray)

      Work only with MyArrayCleared from this point onward. Why? My guess is, when appending the old elements, along with a faulty one because of item interaction, it simply skips adding the InvalidForm element to the new array, making it ctd-proof. Once you are done with what you needed, simply Clear(MyArray) and Delete(MyArrayCleared). This is by no means the only way to handle them after your script is done using them, just an example, but the main point is: if your array became corrupted/is storing a faulty invalid element, simply append it all to the new one to get rid of the error-causers. Cheers to whoever this might be helpful to!
    4. DlinnyLag
      DlinnyLag
      • member
      • 14 kudos
      In the source script of DS:FormArray it says that it "return default value if" and I'm a bit lost, since it's not a false
      Because the default value for Forms is None.
      In if statement None can be implicitly casted to false in certain cases. But it doesn't mean that None is equal to false

      I guess it's not a -1 either?
      There is no such default value. For int type the default value is 0.


      This produces the crash upon item pickup and iteration to delete the items spawned.
      F4DS should not crash. Perhaps you passed None value to subsequent funcations calls and they crashed.
      If you have a chance to share crash details please share them. If you use pre next-gen version of Fallout 4 then you can use Buffout 4 to collect crash log.
      Anyway, I will test, perhaps the default value I return is not correctly initialized.
      UPD: crash reproduced. Will fix it.
      UPD2 : Fixed.

      My guess is, when appending the old elements, along with a faulty one because of item interaction, it simply skips adding the InvalidForm element to the new array
      Correct. But I'm not sure that it is a desired behavior. I suppose I should give an option how to handle invalid forms in that cases. There are 2 options - skip and replace by default value (i.e. None).
    5. DeadlyStr1ke
      DeadlyStr1ke
      • member
      • 36 kudos
      Just tested the new version and indeed, the problem is fixed. Thank you for providing such a quick fix for this amazing utility!
  3. vwtrotsky
    vwtrotsky
    • premium
    • 118 kudos
    Hi, will it be updated for 1.10.984?
    1. DlinnyLag
      DlinnyLag
      • member
      • 14 kudos
      Yes, it will. Approx ETA is about 2-3 weeks
      I need some time to setup new development environment. 
    2. vwtrotsky
      vwtrotsky
      • premium
      • 118 kudos
      Any news?
    3. DlinnyLag
      DlinnyLag
      • member
      • 14 kudos
      It was delayed, unfortunately.
      Code changes are done, will be released after tests.
  4. yoyoma3010
    yoyoma3010
    • premium
    • 92 kudos
    Hi, I'm a novice scripter.

    I assume to make use of the arrays without element limits, I need to call the functions like DS:FormArray.Create(Keyword).

    I'd want to turn an native or F4SE function that returns an array (for example, GetInventoryItems() ), and transfer the all of the array's content to this mod's usage of arrays.

    I guess the issue would be that the native or F4SE function wouldn't be able to generate more than the array limit, so I still wouldn't get the entire inventory of the player for instance.

    Is there something I'm missing, or are the native functions that return arrays still going to be limited even if I try to transfer them over to this mod's arrays?
    1. DlinnyLag
      DlinnyLag
      • member
      • 14 kudos
      Actually, GetInventoryItems function introduced by F4SE should not be limited as the limit is applied when .Add and .Insert functions are called. GetInventoryItems doesn't call these functions internally.

      You should be able to get all inventory items by calling GetInventoryItems. But you will not be able to fully copy elements from array returned by GetInventoryItems to another array in Papyrus script, as you will need to use .Add or .Insert function.

      You are able to transfer all elements of array returned by GetInventoryItems to DS:FormArray as there is no limits. But most likely you don't need it - you already have native Papyrus array populated.

      About other native functions that may generate some arrays. I don't know implementation details, so probably 128 limit is applied in them, probably not.
      And additional note - LL FourPlay plugin has an option to alter this 128 limit. You can take a look to source codes to get implementation details. This F4SE plugin alters .Add and .Insert functions to use configurable limit instead of hardcoded.
  5. DanielIllah2
    DanielIllah2
    • supporter
    • 3 kudos
    This tool is looking very useful, thanks for giving access to it.

    I'm pretty new to this so maybe you'd like to explain this code comment:

    "identifier is the object id that must be used in subsequent calls of functions from this script"

    Is the keyword in the arguments what you mean by 'Object ID'? Am I creating and assigning a keyword to whatever is processed, in real time as I process it with these functions, or must I create a keyword before I run them?
    1. DlinnyLag
      DlinnyLag
      • member
      • 14 kudos
      Is the keyword in the arguments what you mean by 'Object ID'?
      Exactly. Keyword is used as identifier of collection. 
      You could take a look to the example tests script in downloading area to get the idea.

      or must I create a keyword before I run them?
      Yes, you have to declare Keyword in your ESP/ESL/ESM and use it in your script.
    2. DanielIllah2
      DanielIllah2
      • supporter
      • 3 kudos
      Thanks. :)
  6. ImperiusGaunt
    ImperiusGaunt
    • member
    • 23 kudos
    I'm no coder but this look promising! Great work!
  7. ID8879488948574
    ID8879488948574
    • premium
    • 16 kudos
    YEEEEEEEEEEEEEEEEEEEEEY! Just what I desperately need!!

    How are 128+ arrays implemented. F4SE plugin?

    Maps = Dictionaries? What about search (and respectively search time)? How collision-prone they are?

    Dreamland - linked lists, pointers, trees (:
    1. DlinnyLag
      DlinnyLag
      • member
      • 14 kudos
      How are 128+ arrays implemented. F4SE plugin?
      Sure. But it is not arrays with familiar syntax.
      Maps = Dictionaries?
      yes
      What about search (and respectively search time)?
      Hashtables. I intend to use standard unordered_map

      Dreamland - linked lists, pointers, trees (:
      Linked lists will be useless. Marshaling will eliminate any benefits of linked list. Use arrays - they will be fast enough.
      No pointers - Papyrus is a script language. 
      Trees - if you could share any usefull use case... Probably it should be better to not provide generic solution. Just implement specific case as F4SE plugin. Different plugin %)
    2. ID8879488948574
      ID8879488948574
      • premium
      • 16 kudos
      Sure. But it is not arrays with familiar syntax.

      So long as it allows for the data types that Papyrus does and supports:

      • Random access (i.e. access by an index directly)
      • Extending the array (Papyrus .Add)
      • Removing an element of an array (Papyrus .Remove)
      • Searching (linear, as Papyrus .Find)

      I'm content. If you pull off basic array operations, like these (in the order of importance)

      • Sorting
      • Intersection (getting the common set of elements between the arrays)
      • Joining (i.e. gluing together arrays of same data types)
      • Filtering (say with a callback string to use via CallFunction / CallGlobalFunction)


      It's be even better. I had to implement all of those in Papyrus for several mods already. Now.. if you can pull off stuff like MapReduce.. well (:
      Do you not yet allow Refs in the arrays (i.e. not Forms but descendant types of ObjectReference)? That's be much more critical than Forms for the reason FormList-s exist and they do not have the 128 elements limitation

      Hashtables. I intend to use standard unordered_map

      Awesome! Looking forward to see what operations can be supported. Finally will get rid of my clumsy "Map" implementation with the linear search time via Structs.

      Trees - if you could share any usefull use case..
      Why.. BTREE indexing of course. Anything where searching speed is much more important than initial indexing and rebuilding the DS on inserts/deletions will serve as a good use-case. As it happens, I recently made a mod where I manually index and create a flat array of elements out of a large list to speed up the search on every entry - I reckon it'd be even faster if I build a tree instead, but doing it in Papyrus is.. you get the idea.
    3. DlinnyLag
      DlinnyLag
      • member
      • 14 kudos
      So long as it allows
      It is

      Sorting
      Got it. But for Forms I can see the only one way - prepare predefined set of comparers. Do you have any suggestions about what kind of comparisons could be useful? 

      Intersection 
      It could be useful for Sets. But it is possible to "subtract" one array from another in same way as for sets.

      Joining 
      I intend to add .AddRange() for Arrays and Sets

      Filtering 
      No way. I will not prepare solution when synchronous calls of Papyrus code will be required. 

      Do you not yet allow Refs in the arrays (i.e. not Forms but descendant types of ObjectReference)?
      ObjectReference is derived from Form. So you can use ObjectReferences too.

      FormList-s exist and they do not have the 128 elements limitation
      Arrays mostly intent for integral types as Papyrus hasn't 128+ solution for them. Forms support added just because it is easy.

      BTREE indexing of course.
      I guess it is intent to use b-tree for search in "map". You will not need b-tree if you will have hash-table. Performance difference will be insignificant, but hash-table will win %)

      if you can pull off stuff like MapReduce
      Could you explain what do you mean? I don't think that you want MapReduce as we haven't Big Data in game %)
    4. ID8879488948574
      ID8879488948574
      • premium
      • 16 kudos
      Got it. But for Forms I can see the only one way - prepare predefined set of comparers. Do you have any suggestions about what kind of
      comparisons could be useful?

      Well, it looks like you've already answered that part of the question with the hard "NO":

      No way. I will not prepare solution when synchronous calls of Papyrus code will be required.

      as for forms I'd just use callback comparators.

      Could you explain what do you mean? I don't think that you want MapReduce as we haven't Big Data in game %)
      Sorry, this was meant to be Map & Reduce:

      https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html#map-java.util.function.Function-
      https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html#reduce-T-java.util.function.BinaryOperator-

      which again, I guess - following the "NO" to the filtering has the answer already.
    5. DlinnyLag
      DlinnyLag
      • member
      • 14 kudos
      You are right. Papyrus callbacks will not be supported. At least not with this engine. It is possible, in general, but I belive that it will be easier to write my own Fallout 4 than trying to alter existing engine %)
      The main problem - called Papyrus script might never be finished.

      Regarding Forms sorting.
      I found it useful to sort forms by FormId, so Forms from different mods could be sorted in same order as mods.
      But probably you have another options in mind how to compare Forms. 
    6. ID8879488948574
      ID8879488948574
      • premium
      • 16 kudos
      You are right. Papyrus callbacks will not be supported. At least not with this engine. It is possible, in general, but I belive that it will
      be easier to write my own Fallout 4 than trying to alter existing engine
      %)
      The main problem - called Papyrus script might never be finished.


      If that's the problem then I'm not sure I get it. Any callback/method/whatnot has a risk of never finishing and you can rigorously show that's the case in any language so long as we're talking about the basic Turing machine (aka https://en.wikipedia.org/wiki/Halting_problem )

      So I'd look at this other way - if someone engineered their own demise - let it be on them, it shall not be a reason for preventing a very useful functionality to be there. As some guard rails you can think of something like:

      • Only string callback definitions (i.e. for CallFunction/CallGlobalFunction)
      • Only Global functions are allowed (to make sure there's no state dependency)
      • Only Const scripts are allowed (same reason)
      etc. Of course - I'm not trying to tell you what to do, but the genuine risk is there for any callback. You can just define a hard limit on processing and kill the thread if say after 30 seconds the array was still not fully processed. Or not (let it break - after all, it was the script author who did it) - I like to have a freedom to shoot myself in the foot :p
    7. DlinnyLag
      DlinnyLag
      • member
      • 14 kudos
      I've got you point. Usage of pure functions without side effects could reduce probability of infinite time of method call. No guarantees, however.
      But forward advance guarantee is not the only problem. I need to handle save/load process and save/restore internal iteration state somehow. It seems feasible, but complex. As work around, I can execute iterations within critical section to avoid necessity to save/restore state but I can't predict will it lead to deadlocks when engine will initiate save process. Not at this moment.

      You can just define a hard limit on processing and kill the thread if say after 30 seconds the array was still not fully processed.

      Oh no %)
      I have no idea what exactly will happens with engine if thread will be terminated. How it will affect Papyrus script execution FSM? I'm pretty sure engine is not ready to handle unexpected thread termination. We will be lucky if it will just cause a crash, without any data damaging %)

      I guess I can add ForEach() functionality in "fire and forget" manner. Actully, I have plans to implement iterators.
      But map(), reduce(), sort() or any other algoritm that will require result from script function... no. 
      Anyway, Papyrus callbacks is not a priority. Maybe, at some moment I will back to this idea, but not in nearest future.

      I share with you the lust for scripted callbacks, but it looks like it's not worth the time.
    8. ID8879488948574
      ID8879488948574
      • premium
      • 16 kudos
      Ok, I got your point. Stability >>> Functionality, here I fully support you.

      Now, can we turn our eyes to more basic things? (: I'm looking at you, Strings
      Since you already did such an awesome job with the DS, maybe you could expand the functionality for Strings too? LL_fourplay has some methods of working with strings, but they are fairly limited. What I'd love to see is:

      • Getting string length (duh)
      • String replacement (one string in another)
      • Regex? (oh wow..)

      Apart from that I reckon there's very lacking functionality in the game to displaying the strings to the user. I know it isn't exactly the theme of this mod - but since it looks like you have a good grasp on F4SE plugins, maybe you can consider this: how about expanding the standard Debug.Notification call with at least notification duration? That method is the only currently existing non-intrusive way to show the strings in-game, but it really doesn't work for longer strings as the duration is static and is just too short.
    9. DlinnyLag
      DlinnyLag
      • member
      • 14 kudos
      I'm looks at you, Strings
      It is my next topic to work on after DS.
      I will need string functions to implement ideas I want in  Know Your Friend

      That method is the only currently existing non-intrusive way to show the strings in-game
      I'm not sure about "non-intrusive", but you could take a look to Console Util
    10. ID8879488948574
      ID8879488948574
      • premium
      • 16 kudos
      It is my next topic to work on after DS.
      I will need string functions to implement ideas I wanted in  Know Your Friend

      +1, great

      I'm not sure about "non-intrusive", but you could take a look to Console Util

      Well, something that'd be for the user of the mod, not the modder themselves. LL_fourplay already has PrintConsole to output into the console.
      Biggest issue with strings and interactions there is that Fallout4 UI is just not well-suited for them. So any output of the information to the user in strings form is a headache. Some mods do this with HUD Framework and SWF widgets but those aren't flawless and require to actually build an Action Script plugin + of course have a hard dependency on HUD Framework. It's too much hassle for just showing the strings.

      So here we go - some mods (like in my case) are just like "sigh.. whatever, let's just use Debug.Notification even if it flashes in a second and user struggles to see the info", some are "okay, here's your Debug.MessageBox" and some just avoid strings altogether (: If I knew F4SE plugin creation, I'd do it long ago when working with strings.
  8. YouDoNotKnowMyName
    YouDoNotKnowMyName
    • member
    • 6 kudos
    Wohooooo!
    This is going to be soooo useful!

    Technically if you can add to the arrays it wouldn't be an array it would be a vector ...
    Anyway, just what I need!

    THANKS SO MUCH!
    1. DlinnyLag
      DlinnyLag
      • member
      • 14 kudos
      Technically if you can add to the arrays it wouldn't be an array it would be a vector ...
      Papyrus arrays allows to add new elements and named arrays, not vectors. I don't want to confuse modders with new naming.
      And yes, vectors are used under the hood.
  9. niston
    niston
    • premium
    • 464 kudos
    The 128 elements limit in arrays is simply enforced by the Papyrus VM on new() and .Add().
    It's possible to patch the limit already and create arrays of arbitrary size, LL_FourPlay plugin (comes with AAF) has support for this.
    1. DlinnyLag
      DlinnyLag
      • member
      • 14 kudos
      So this is the reason to use LL_FourPlay, isn't it? I understand that I can use code from  LL_FourPlay in my plugin, but I do not see a way how to not interfere with LL_FourPlay. So I decided to provide arrays API in another way. It is up to modders what arrays to use.
      Additionnaly, I can not be sure that game code doesn't have assumptions that limit is 128 exactly. I just don't want to risk.