universalid

Fast, recursive, optimized, URL-Safe serialization for any Ruby object

370

Ruby

Universal ID

Fast, recursive, optimized, URL-Safe serialization for any Ruby object

Universal ID leverages both MessagePack and Brotli (a combo built for speed and best-in-class data compression).
When combined, these libraries are up to 30% faster and within 2-5% compression rates compared to Protobuf. ↗

Universal ID introduces a paradigm shift that enables straightforward simple solutions ↗ for a variety of complex problem domains.

[!TIP]
All the code examples below can be tested on your local machine. Just clone the repo (↑or use Gitpod above↑) and run bin/console to begin exploring.
Don’t forget to execute bundle first to ensure all dependencies are up to date. Happy coding!

URI::UID
Supported Data Types
Options
Advanced Usage
Sponsors
License

URI::UID

Universal ID introduces a new URI defintion that can recursively serialize any Ruby object into an URL-safe string
which can be safely transported via HTTP.

[!NOTE]
The payload is optimized to be as small as possible… especially notable with large objects.

The best part: The API is simple.

data = :ANY_OBJECT_YOU_CAN_IMAGINE

uid = URI::UID.build(data)
#<URI::UID payload=Cw6AxxoAQU5ZX09CSkVDVF9ZT1VfQ0FOX0lNQ..., fingerprint=CwWAkccHf6ZTeW1ib2wD>

uid.payload
"Cw6AxxoAQU5ZX09CSkVDVF9ZT1VfQ0FOX0lNQUdJTkUD"

uid.fingerprint
"CwWAkccHf6ZTeW1ib2wD"

uri = uid.to_s
"uid://universalid/Cw6AxxoAQU5ZX09CSkVDVF9ZT1VfQ0FOX0lNQUdJTkUD#CwWAkccHf6ZTeW1ib2wD"

parsed = URI::UID.parse(uri)
#<URI::UID payload=Cw6AxxoAQU5ZX09CSkVDVF9ZT1VfQ0FOX0lNQ..., fingerprint=CwWAkccHf6ZTeW1ib2wD>

parsed.decode
:ANY_OBJECT_YOU_CAN_IMAGINE

# it's also possible to parse the payload by itself

parsed = URI::UID.from_payload(uid.payload)
#<URI::UID payload=Cw6AxxoAQU5ZX09CSkVDVF9ZT1VfQ0FOX0lNQ..., fingerprint=CwWAkccHf6ZTeW1ib2wD>

parsed.decode
:ANY_OBJECT_YOU_CAN_IMAGINE

Supported Data Types

Primitive Types

Universal ID supports most native Ruby primitives:

NilClass
BigDecimal
Complex
Date
DateTime
FalseClass
Float
Integer
Range
Rational
Regexp
String
Symbol
Time
TrueClass

You can use Universal ID to serialize individual primitives, but this actually serves as the foundation for more advanced use-cases.

uri = URI::UID.build(:demo).to_s
#=> "uid://universalid/iwKA1gBkZW1vAw#CwWAkccHf6ZTeW1ib2wD"

uid = URI::UID.parse(uri)
#=> #<URI::UID payload=iwKA1gBkZW1vAw, fingerprint=CwWAkccHf6ZTeW1ib2wD>

uid.decode
#=> :demo

Composite Types

Composite (or complex, compound, etc.) datatype support is where things start to get interesting.
Universal ID supports the following native Ruby composite datatypes:

Array
Hash
OpenStruct
Set
Struct

array = [1, 2, 3, [:a, :b, :c, [true]]]

uri = URI::UID.build(array).to_s
#=> "uid://universalid/iweAlAECA5TUAGHUAGLUAGORwwM#iwSAkccGf6VBcnJheQM"

uid = URI::UID.parse(uri)
#=> #<URI::UID payload=iweAlAECA5TUAGHUAGLUAGORwwM, fingerprint=iwSAkccGf6VBcnJheQM>

uid.decode
#=> [1, 2, 3, [:a, :b, :c, [true]]]

uid.decode == array
#=> true

hash = {a: 1, b: 2, c: 3, array: [1, 2, 3, [:a, :b, :c, [true]]]}

uri = URI::UID.build(hash).to_s
#=> "uid://universalid/CxKAhNQAYQHUAGIC1ABjA8cFAGFycmF5lAECA5TUAGHUAGLUAGORwwM#CwS..."

uid = URI::UID.parse(uri)
#=> #<URI::UID payload=CxKAhNQAYQHUAGIC1ABjA8cFAGFycmF5lAECA..., fingerprint=CwSAkccFf6RIYXNoAw>

uid.decode
#=> {:a=>1, :b=>2, :c=>3, :array=>[1, 2, 3, [:a, :b, :c, [true]]]}

uid.decode == hash
#=> true

Book = Struct.new(:title, :author, :isbn, :published_year)
book = Book.new("The Great Gatsby", "F. Scott Fitzgerald", "9780743273565", 1925)

uri = URI::UID.build(book).to_s
#=> "uid://universalid/G2YAoGTomv9tT1ilLRgVC9vIpmuBo-k84FZ0G8-siFMBNsbW0dpBE0Tnm96..."

uid = URI::UID.parse(uri)
#=> #<URI::UID payload=G2YAoGTomv9tT1ilLRgVC9vIpmuBo-k84FZ0G..., fingerprint=CwSAkccFf6RCb29rAw>

uid.decode
#=> #<struct Book title="The Great Gatsby", author="F. Scott Fitzgerald", isbn="9780743273565", published_year=1925>

uid.decode == book
#=> true

Extension Types

The following extension datatypes ship with Universal ID:

ActiveRecord::Base
ActiveRecord::Relation
ActiveSupport::Cache::Entry
ActiveSupport::Cache::Store
ActiveSupport::TimeWithZone
GlobalID
SignedGlobalID

[!NOTE]
Extensions are autoloaded whenever the related datatype is detected.

[!IMPORTANT]
Why Universal ID with ActiveRecord?
ActiveRecord already has GlobalID, a robust library for serializing individual models.
Universal ID covers a much wider range of use cases.

Here are a few reasons you may want to consider Universal ID with ActiveRecord.

New Records:
Universal ID can serialize models that haven’t been saved to the database yet.
Changesets:
Universal ID can serialize ActiveRecord models with unsaved changes, ensuring that even transient states are captured.
Associations:
Universal ID goes beyond single models. It can include associated records, even those with unsaved changes, creating a comprehensive snapshot of complex record states.
Copying/Cloning:
Universal ID supports making copies of records (including associations), making it ideal for duplicating complex datasets.
More Control:
Universal ID gives you control over the serialization process. You can choose which columns to include/exclude, allowing for tailored, optimized payloads to fit your needs.
Queries/Relations:
Universal ID also supports ActiveRecord::Relations, enabling the serialization of complex database queries and scopes.

In summary, while GlobalID excels in its specific use case, Universal ID offers more power for use-cases that involve unsaved records, complex associations, data cloning, and database queries.

# setup some records
campaign = Campaign.create(name: "My Campaign")
email = campaign.emails.create(subject: "First Email")
attachment = email.attachments.create(file_name: "data.pdf")

# ensure associations are loaded so they can be included in an UID
campaign.emails.load
campaign.emails.each { |e| e.attachments.load }

# make some unsaved changes
email.subject = "1st Email"

# add an unsaved record
campaign.emails.build(subject: "2nd Email")

# introspection
campaign.emails.size #=> 2
campaign.emails.loaded? #=> true
campaign.emails.last.new_record? #=> true

options = {
  include_changes: true,
  include_descendants: true,
  descendant_depth: 2
}

uri = URI::UID.build(campaign, options).to_s
#=> "uid://universalid/GxYBYGT6_Xn_OrelIDRWhQQgvbS5gQxV7EJKe3paIiEFmEEc1gLKw8Pl2-k..."

uid = URI::UID.parse(uri)
#=> #<URI::UID payload=GxYBYGT6_Xn_OrelIDRWhQQgvbS5gQxV7EJKe..., fingerprint=CwuAkscJf6hDYW1wYWlnbtf_ReuZnGWeG5MD>

decoded = uid.decode
#=> "#<Campaign id: 13, name: \"My Campaign\" ...>

decoded == campaign
#=> true

# introspection
decoded.emails.size #=> 2
decoded.emails.loaded? #=> true
decoded.emails.first.changed? #=> true
decoded.emails.first.changes #=> {"subject"=>["First Email", "1st Email"]}
decoded.emails.last.new_record? #=> true
decoded.save #=> true
decoded.emails.last.persisted? #=> true

Custom Types

Universal ID is extensible, enabling you to register your own datatypes with custom serialization rules.
Simply convert the required data to a Ruby primitive or composite value.

# create a custom type
class UserSettings
  attr_accessor :user_id, :preferences

  def initialize(user_id, preferences = {})
    @user_id = user_id
    @preferences = preferences
  end
end

# register the custom type with Universal ID
UniversalID::MessagePackFactory.register(
  type: UserSettings,
  packer: ->(user_preferences, packer) do
    packer.write user_preferences.user_id
    packer.write user_preferences.preferences
  end,
  unpacker: ->(unpacker) do
    user_id = unpacker.read
    preferences = unpacker.read
    UserSettings.new user_id, preferences
  end
)

# create an instance of the custom type
settings = UserSettings.new(1,
  theme: "dark",
  notifications: "email",
  language: "en",
  layout: "grid",
  privacy: "private"
)

# serialize the custom type
uri = URI::UID.build(settings).to_s
#=> "uid://universalid/G1QAQAT-c_cO7qJcAk-TtsAiadci_IA5xoH7NV3bYttEww7xuUkzasu2HEO..."

# deserialize the custom type
uid = URI::UID.parse(uri)
#=> #<URI::UID payload=G1QAQAT-c_cO7qJcAk-TtsAiadci_IA5xoH7N..., fingerprint=CwiAkccNf6xVc2VyU2V0dGluZ3MD>

uid.decode
=> #<UserSettings:0x000000011d0deb20 @preferences={:theme=>"dark", :notifications=>"email", :language=>"en", :layout=>"grid", :privacy=>"private"}, @user_id=1>

Options

Universal ID supports a small, but powerful, set of options used to “prepack” the object before it’s packed with MessagePack.
These options instruct Universal ID on how to prepare the object for serialization.

prepack:
  # ..........................................................................................................
  # A list of attributes to exclude (for objects like Hash, OpenStruct, Struct, etc.)
  # Takes prescedence over the`include` list
  exclude: []

  # ..........................................................................................................
  # A list of attributes to include (for objects like Hash, OpenStruct, Struct, etc.)
  include: []

  # ..........................................................................................................
  # Whether or not to include blank values when packing (nil, {}, [], "", etc.)
  include_blank: true

  # ==========================================================================================================
  # Database records
  database:
    # ......................................................................................................
    # Whether or not to include primary/foreign keys
    # Setting this to `false` can be used to make a copy of an existing record
    include_keys: true

    # ......................................................................................................
    # Whether or not to include date/time timestamps (created_at, updated_at, etc.)
    # Setting this to `false` can be used to make a copy of an existing record
    include_timestamps: true

    # ......................................................................................................
    # Whether or not to include unsaved changes
    # Assign to `true` when packing new records
    include_changes: false

    # ......................................................................................................
    # Whether or not to include loaded in-memory descendants (i.e. child associations)
    include_descendants: false

    # ......................................................................................................
    # The max depth (number) of loaded in-memory descendants to include when `include_descendants == true`
    # For example, a value of (2) would include the following:
    #   Parent > Child > Grandchild
    descendant_depth: 0

Options can be applied whenever creating a UID.

hash = { a: 1, b: 2, c: 3 }

uri = URI::UID.build(hash, exclude: [:b]).to_s
#=> "uid://universalid/CwSAgtQAYQHUAGMDAw#CwSAkccFf6RIYXNoAw"

uid = URI::UID.parse(uri)
#=> #<URI::UID payload=CwSAgtQAYQHUAGMDAw, fingerprint=CwSAkccFf6RIYXNoAw>

uid.decode
#=> {:a=>1, :c=>3}

[!NOTE]
Options can be passed in structured or flat format.

It’s also possible to register frequently used options.

# app/config/changed.yml
prepack:
  include_blank: false

  database:
    include_changes: true
    include_descendants: true
    descendant_depth: 2

UniversalID::Settings.register :changed, File.expand_path("app/config/changed.yml", __dir__)
uid = URI::UID.build(record, UniversalID::Settings[:changed])

Advanced Usage

Fingerprinting

Each UID is fingerprinted as part of the serialization process.

Fingerprints are comprised of the following components:

Class (Class) - The encoded object’s class
Timestamp (Time) - The mtime (UTC) of the file that defined the object’s class

Fingerprints provide a simple mechanism to help manage data format versions… minimizing the need for custom versioning solutions.
Whenever the class definition changes, the mtime updates, resulting in a different fingerprint.
This is especially useful in scenarios where the data format evolves over time, such as in long-lived applications.

uid = URI::UID.build(campaign)

uid.fingerprint
#=> "CwuAkscJf6hDYW1wYWlnbtf_ReuZnGWeG5MD"

uid.fingerprint(decode: true)
#=> [Campaign(id: integer, ...), <Time>]

[!NOTE]
The timestamp or mtime is determined the moment a UID is created.

[!TIP]
Fingerprints can help you maintain consistency and reliability when working with serialized data over time.
While fingerpint creation is automatic and implicit, usage is optional… ready whenever you need it.

Copy ActiveRecord Models

Make a copy of an ActiveRecord model (with loaded associations).

campaign = Campaign.first

# ensure desired associations are loaded so they can be included in an UID
campaign.emails.load
campaign.emails.each { |e| e.attachments.load }

# introspection
campaign.id #=> 1
campaign.emails.map(&:id) #=> [1, 2]
campaign.emails.map(&:attachments).flatten.map(&:id)
#=> [1, 2, 3, 4]

# setup options for copying
options = {
  include_blank: false,
  include_keys: false,
  include_timestamps: false,
  include_descendants: true,
  descendant_depth: 2
}

uri = URI::UID.build(campaign, options).to_s
#=> "uid://universalid/G7kAIBylMxZa7MouY3gUqHKkIx3hk4s8NT5xWwQsDc7lKUkGWM4DHsCxQZK..."

uid = URI::UID.parse(uri)
#=> #<URI::UID payload=G7kAIBylMxZa7MouY3gUqHKkIx3hk4s8NT5xW..., fingerprint=CwuAkscJf6hDYW1wYWlnbtf_ReuZnGWeG5MD>

copy = uid.decode
#=> #<Campaign:0x00000001135c7448 id: nil, name: "My Campaign", ...>

copy == campaign
#=> false

# introspection
copy.new_record? #=> true
copy.id #=> nil
copy.emails.map(&:id) #=> [nil, nil]
copy.emails.map(&:attachments).flatten.map(&:id)
#=> [nil, nil, nil, nil]

# create the copy (new records) in the database
copy.save #=> true

[!TIP]
If you don’t need a URL-Safe UID, you can use UniversalID::Packer to speed things up a bit.

packed = UniversalID::Packer.pack(campaign, options)
copy = UniversalID::Packer.unpack(packed)
copy.save

ActiveRecord::Relations

Universal ID also supports ActiveRecord relations/scopes.
You can easily serialize complex queries into a portable and sharable format.

relation = Campaign.joins(:emails).where("emails.subject LIKE ?", "Flash Sale%")

uri = URI::UID.build(relation).to_s
#=> "uid://universalid/G90EQCwLeEP1oQtHFksrdN5YS4ju5TryFZwBJgh2toqS3SKEVSl1FoNtZjI..."

uid = URI::UID.parse(encoded)
#=> #<URI::UID payload=G90EQCwLeEP1oQtHFksrdN5YS4ju5TryFZwBJ..., fingerprint=CxKAkscXf7ZBY3RpdmVSZWNvcmQ6OlJlbGF0a...>

decoded = uid.decode

# introspection
decoded == relation #=> true
decoded.is_a? ActiveRecord::Relation #=> true
decoded.loaded? #=> false

# run the query
campaigns = decoded.load

[!NOTE]
Universal ID clears cached data within the relation before encoding. This minimizes payload size while preserving the integrity of the underlying query.

SignedGlobalID

Features like signing (to prevent tampering), purpose, and expiration are provided by SignedGlobalIDs.
These features (and more) will eventually be added to Universal ID, but until then…
simply convert your UID to a SignedGlobalID to add these features to any Universal ID.

data = OpenStruct.new(name: "Demo", value: "Example")

sgid = URI::UID.build(data).to_sgid_param(for: "purpose", expires_in: 1.hour)
#=> "eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaEpJZ0plQTJkcFpEb3ZMM1Z1YVhabGNuTmhiQzFwWkM5V..."

uid = URI::UID.from_sgid(sgid, for: "purpose")
#=> #<URI::UID payload=Cw-Axxx-gtYAbmFtZaREZW1vxwUAdmFsdWWnR..., fingerprint=ixqAkscof9kmVW5pdmVyc2FsSUQ6OkV4dGVuc...>

decoded = uid.decode
#=> #<OpenStruct name="Demo", value="Example">

# a mismatched purpose returns nil... as expected
URI::UID.from_sgid(sgid, for: "mismatch")
#=> nil

License

The gem is available as open source under the terms of the MIT License.

universalid

Universal ID

Fast, recursive, optimized, URL-Safe serialization for any Ruby object

Table of Contents

URI::UID

Supported Data Types

Primitive Types

Composite Types

Extension Types

Custom Types

Options

Advanced Usage

Fingerprinting

Copy ActiveRecord Models

ActiveRecord::Relations

SignedGlobalID

Sponsors

License