Fast(er) Set creation

Set creation can be quite slow for large sets (> 15000ish string items). If input sequence to create the set is sorted then some optimizations can be applied. For even larger unordered sets (> 30000ish string items) it can be faster doing an up front sort on the data, and then using the Set creation method as described. 1) Set.union is very fast when the greatest element in one of the sets is less than the smallest element in the other; basically becoming an O(1) operation. And Set.add is faster for smaller sets than larger sets, given O(log2 n) of the add operation. So when we have ordered data, makings lots of smaller sets from the stream and union-ing them together can provide a performance boost. 2) On top of the method described in (1), because all the sets are immutable inputs and outputs, then they can be partitioned off onto Tasks to perform the set creation in parallel. If you are using Newtonsoft's Json.net, then provided is a JsonConverter that can be added to the serializer to use this for Set creation like: serializer.Converters.Add Newtonsoft.fastFSharpSetConverter

Posted: 9 years ago by manofstick