Models and collections are scary

General
#1

Is it just me or are models and collections scary complicated?

maybe it’s because the data I’m trying to get uses uuid’s rather than sequential primary keys but the amount of wiring needed seems to be a lot; id conversions, transformations, and storage interfaces that require implementing a dozen methods, it feels like a lot for something that should be much simpler. perhaps I’m over thinking things and doing more work than i should? am i?

in contrast regular calls are really easy, I’ve implemented a list call method which does the trick for now but it’s bugging me that i should in fact be using models and collections to get lists of information.

#2

Models and collections per se are not that complicated. They are just flat json objects, or flat arrays, returned in get requests. Kind of like a call method request.

But I agree that the higher level go-res/store API is more complex (and more powerful). But then again, it is by no means required to use. Most Resgate users don’t use it, as it is a fairly new addition.

Would you like help understanding the store API? (I really should write a blog post about it)
Or help understanding how you can implement a list without it?

/Samuel

#3

Yeah I think its the Store which is scary due to lack of documentation. this is the type of data I’m returning, i’d like to use the model so I can publish the on change event and have that automatically updated. Right now Im using call and just recall when I need updates…

{
  "result": {
    "payload": {
      "GACJ5MWKG5NRQ5OVC7XHRWJCFSGDPU2CNLVFGTAEN3JMBH6J7TSR7RSF": {},
      "GASCHM35A2WTWNFSCWV6DM5F2EDS4TGQUKNFAHUDQ6IN27JFCBYPXUAS": {},
      "GB254BYZRZXWS5FVPWXOMOHJ3XPGTURRUQDP3LGBOIP35EW3IEDVWRCV": {},
      "GBNUDUNMBCXY6SSF2CW7KKOKOHIRQEL2MUFS6LQWUYWAL6S5WI5UQ3ED": {},
      "GC2O5JAKMSBHZLEL4O34VJQTQR3NCO4KTGCC4LSIKTXK2XFLSORQI5B7": {},
      "GCG2NTQCWSCSPORVCNEFEM7HBZR7YXSH5HIJV25FEFQAHVEEONPK5XMJ": {},
      "GCI2OE2AAFVK5ZBEBNW2I7M7TAWXLOGNRL6WMEAR7CPAFWQ5UF5NNCXM": {},
      "GCJSDAPMFT5VP46PNEJOHCK3XASBAW55RNQQYNAXZCX7ZIMXO6SWIU2U": {},
      "GCKR5EMYW2APM6Q3IAYB3457CERYU6VVPIXNDHQSY7A2JBKR5IN5LU3H": {},
      "GCQE53G43X7PFWGJUMFGARX5OJHQDN43L54PNLATJ7FRM7GCMC3SIHOL": {},
      "GD2ISV2RCGEDZ7UU3JM2VVCTL6WP6Q2TKFIFBSWMUZS4W4N3ELQUMEN4": {},
      "GD3KDT3OXWBV4BL5RP6KKXLZJLPKKPSBWPSI7ETFS7B3CEBHRSHBHWNY": {},
      "GDGPPMD6LXISY3JHCQZNTX3WDXS4YJ5OFJ56BIZADCBSSYGGW6223352": {},
      "GDY7T5HEA35P5J4WU6LQSFX7ACGWL75X52TG56JF4HEDIDHWNO6YYBTB": {}
    }
  },
  "id": 3
}
#4

also i’m an idiot, one of the reasons I was finding collections and models difficult is because I was expecting them to have methods like the calls so it was short circuiting my brain trying to add handlers… so yeah… i’m an idiot :slight_smile:

#5

In a sense, call requests are the methods of models/collections.

You can have a model fetched with a get request:

get.directory.user.42:

{ "id": 42, "name": "Samuel Jirénius", "role": "guest" }

And you have have call methods on that model:

call.directory.user.42.ban

{ "reason": "No particular reason" }

But did you get it to work?

Your example data (an unordered set of id strings) is easy and can be done kind of like this:

s := res.NewService("example")

s.Handle("myids",
   res.Access(res.AccessGranted),
   res.GetModel(func(r res.ModelRequest) {
      // Fetch your data from the database.
      // I use bool instead of struct{} since a model's values should be
      // primitives and not objects/arrays.
      var data map[string]bool = fetchMyIDStrings()
      r.Model(data)
   }),
)

And to update them:

// Here we use With to get the resource context.
s.With("example.myids", func(r res.Resource) {
    // Add an ID
    newID := "GEKR5EMYW2APM6Q3IAYB3457CERYU6VVPIXNDHQSY7A2JBKR5IN5LU3H"
    addIDToMyDatabase(newID)
    r.ChangeEvent(map[string]interface{}{newID: true})

    // Delete an ID
    deleteID := "GACJ5MWKG5NRQ5OVC7XHRWJCFSGDPU2CNLVFGTAEN3JMBH6J7TSR7RSF"
    deleteIDFromMyDatabase(deleteID)
    r.ChangeEvent(map[string]interface{}{deleteID: res.DeleteAction})
})

Or if you don’t want to send ChangeEvents (maybe because you don’t know exactly what changed), you can also Reset the model, meaning that you tell Resgate to fetch the data again, compare it with its cache, and send updates to all relevant clients.

s.Reset([]string{"example.myids"}, nil)

The Reset event is less prone to race conditions, and easier to work with, but is a bit less efficient.

I’ll explain the Store API some other time :slight_smile:

/Samuel

#6

ah ok that helps a lot.

in it’s simplest form it’s really easy, i however, was not trying the simplest, i was using a handler struct and trying to set the option as a model which is where my brain short wired.

i was trying to determine what you call the method so the model in the struct calls it, like in a call.

When looking at the Store Api there is an interface but that was way more than i wanted to deal with, i just wanted a simple model handler using the setOptions in a struct.

is there a way to associate a method with the model in a struct handler setOptions e.g. r.Get(someMethod)

#7

Not sure what you mean with “associate a method with the model”. Is this what you are after?

package main

import res "github.com/jirenius/go-res"

type MyHandler struct {
	// Stuff that the handler need. Maybe a DB client or something.
}

// SetOption is to implement the res.Option interface
func (h MyHandler) SetOption(rh *res.Handler) {
	rh.Option(
		res.GetModel(h.getModel),     // use a struct method for handling get requests
		res.OnRegister(h.onRegister), // Not needed. Just added to show the OnRegister option
	)
}

// onRegistered is called when the handler is added to a service.
func (o *MyHandler) onRegister(s *res.Service, p res.Pattern, h res.Handler) {
	// s is the service instance
	// p is the full resource pattern ("example.model")
	// h is the handler struct that was registered
}

func (h *MyHandler) getModel(r res.ModelRequest) {
	r.Model(struct {
		Message string `json:"message"`
	}{"Hello, World!"})
}

func main() {
	s := res.NewService("example")
	s.Handle("model",
		res.Access(res.AccessGranted),
		MyHandler{},
	)
	s.ListenAndServe("nats://localhost:4222")
}
#8

:man_facepalming: it’s so obvious now that i see it. Thank you this will make life much easier.

1 Like
#9

First hiccup with models. The token isnt available. I was using that in the list call to determine what to send… it looks like that might be intentional in the sdk, I can see the fork added CID and token to all calls probably for similar reasons why i need it.

I tried the fork both the go-res and resgate changes, it seems like there is still some work needed to get the token to come through on the ModelRequest.

#10

Very intentional indeed!

This enables the caching mechanism, and is one of the reasons behind Resgate’s outstanding performance.

If 1000 clients requests a resource (model or collection), Resgate only need to request it once from the service, and then return the same result to all 1000 clients. And it can cache it for future requests.

Now, if the returned resource could depend on a token or CID, there is no way Resgate could determine if those 1000 clients would get the same data if they have different tokens… and Resgate would have to make 1000 get requests instead of 1. Thus making the caching mechanism close to worthless for logged in clients.

Sooo, that said. How to do user-specific resources?

Quite simple. Whatever information that is required to identify the unique resource should be part of the resource ID (including the query part).

Example

Let’s say we have a movie database service.
And the user/client has a token:

{ "userId": 42, "role":"user", "countryId": "SE" }

If we have a resource (collection) which contains a list of the users favorites movies, those resource ID’s could look like this:

"moviedb.myFavoriteMovies"  // Bad. This would not be unique for each user, and we cannot rely on the token
"moviedb.user.$userId.favoriteMovies" // Good. The userId it is part of the resource name

Another resource giving the top movies. This can also be filtered by country:

"moviedb.topList" // This means top list for _all_ countries
"moviedb.topList?country=SE" // The additional information is provided in the query, instead of relying on token

If the client doesn’t know the userId of the logged in user, you can have a call method that reads it from the token and returns a resource response to the actual resource:


func main() {
	s := res.NewService("moviedb")
	s.Handle("", // Put the call method on the service root
		res.Call("myFavoriteMovies", func(r res.CallRequest) {
			// Parse the userId from the token in the call request
			token := struct {
				UserID string `json:"userId"`
			}
			r.ParseToken(&p)
			// Return a reference to the unique resource for that user.
			// Resgate will fetch this resource and return it in the call response.
			r.Resource(fmt.Sprintf("moviedb.user.%s.favoriteMovies", p.UserID))
		}),
	)
	// ... do more ... such as adding a handler for "user.$userId.favoriteMovies"
}

Hope that clarifies it :slight_smile:

/Samuel

#11

OK that makes sense.

I guess it means that the call is the right solution for what I used it for. We are returning highly sensitive information and performance is less of a concern than security. It is good to know more about the models, when I have less secure information to return I’ll use that instead.

I imagine I could accomplish similar security using the topics with the user id in the topic but don’t want to go down that road where someone could potentially manipulate a topic to get information.

Using the call with the token is the access key which adds the additional layer of security that I like, so not only do we have the access check they can’t even pull the sensitive if the token is wrong .

What I liked about the model is sending the change event to update the information, perhaps there is a need for a middle ground between models and call, skip the caching mechanism in exchange for the security and live updates?

#12

Models/collections are just as secure as calls. Access to a model (as well as access to a call method) is always granted explicitly. It is not like with old style REST API where you can miss the access control check. With the RES protocol, access MUST be granted in the separate access request, or else you will never see that data.

With the 1000-clients example, there will only be 1 get request, but Resgate will send 1000 access requests. The benefit is that, validating access for 1000 tokens is always cheaper than loading, encoding, and transmitting the data 1000 times (and you still need to validate access)…

Example

Let’s return to the moviedb example resource: moviedb.user.$userId.myFavorites

Let’s assume you should only have access to moviedb.user.42.myFavorites if you actually are user 42. This is how to do it:

func main() {
	s := res.NewService("moviedb")
	s.Handle("user.$userId.favoriteMovies",
		// This handler handles Get requests and Call requests, like adding or removing favorite movies.
		favoriteMoviesHandler{},
		// Access handler validates if a token can get access to the resource
		res.Access(func(r res.AccessRequest) { 
			// Parse the userId from the token in the access request
			token := struct {
				UserID string `json:"userId"`
			}
			r.ParseToken(&token)
			// Clients only get access to the resource if they have a token with a
			// matching userId. Otherwise we deny access.
			// That means, only userId=42 can access moviedb.user.42.favoriteMovies
			if token.UserID == r.PathParam("userId") {
				r.AccessGranted()
			} else {
				r.AccessDenied()
			}
		}),
	)
}
#13

And, I could add: access to a resource (model/collection) can also be revoked in real-time.

If a client connection gets a new token (maybe because the user’s role changed from "moderator" to "user". Or maybe because the user logged out), all resources currently subscribed by that client will be set on “pause” until Resgate has reconfirmed that the client still has access to the resources using the new token.

If the new token does not pass the access request, the resource will instantly be unsubscribed by Resgate, so that the user will not receive any more events/updates on that resource.

So, security is no problem for Resgate’s models :slight_smile:
Actually, this security feature is what made SAAB Group look at using Resgate. They needed (among other things) real-time access revocation of sensitive information and events.

/Samuel

#14

as always thank you for the detailed explanation . I’m learning a lot from these discussions.

I’m probably a corner case then. For our uses in this case (returning user data) i would still need access to the token in the model.

our user information is encrypted by default.

The signed token is the only way to get information from our data store, even if we approve the request in the access check, the jwt we set in the cid is the only token our data store will decrypt the users information for with the exception of a root token which is physically stored in a safe. this prevents anyone from accessing the data exect the rightful owner.

without this extra step, the service retrieving the data would have access to the unencrypted data and potentially be at risk of exposure by someone with access to that service or it’s credentials.

#15

That changes things.

A bit of a corner case, but a very interesting and useful one.
For now, the call request is the only valid option.

But I can’t help but to think of how the protocol could be extended to support such use cases.
How to make Resgate know that the resource is encrypted, requiring the token to decrypt?

It would probably include the service responding to the initial get request with a result that that says “secure”. And this would trigger Resgate to send a second, secure, get request containing CID and Token for each requesting client.

Anyway. Thanks for sharing! :slight_smile:

#16

so I got the nerve to try using a Model again and failed. seems like I cant return complex objects.

I was trying to return something along the lines of

{
  "self": {"href":"https://www.example.com"},
  "prev": {"href":"https://www.example.com?somequery"},
  "next": {"href":"https://www.example.com?somequery"}, 
  "embedded": {
       "records": [1,2,3,4,5,6,etc.....]
   }

which resulted in this error

Interanl error: Internal error: invalid value

is this because of the issues with nested objects?

#17

It is indeed about how nested resources work.

Models (and collection) values can only be primitives (string, number, boolean, null) or resource references.

Yes, it is different than the static nested JSON blobs that is common with REST API’s. With Resgate, it is encouraged to avoid unnecessary nesting (in your example, that “embedded” might be considered unnecessary). Each json object/array will also be represented by a separate model/collection, linked together using resource references.

In your example, it might look like this:

// ParseInt parses a string to int, or returns 0 on error.
func ParseInt(str string) int {
	if str != "" {
		if v, err := strconv.Atoi(str); err == nil {
			return v
		}
	}
	return 0
}

func main() {
	s := res.NewService("example")

	s.Handle("search",
		// This model only provides links.
		// The actual query results is in the collection below.
		res.GetModel(func(r res.ModelRequest) {
			start := ParseInt(r.ParseQuery().Get("start"))

			r.Model(struct {
				// Instead of string, you can soon use res.SoftRef which will
				// correctly translate into hrefs by Resgate
				Self string `json:"self"`
				Prev string `json:"prev"`
				Next string `json:"next"`
				// Reference to the actual query result records
				Records res.Ref `json:"records"`
			}{
				// Links to self/prev/next. Should add boundry checks
				fmt.Sprintf("example.search?start=%d", start),
				fmt.Sprintf("example.search?start=%d", start-10),
				fmt.Sprintf("example.search?start=%d", start+10),
				// Link to the records collection
				res.Ref(fmt.Sprintf("example.search.records?start=%d", start)),
			})
		}),
	)
	// The records collection resource that actually does the database query.
	s.Handle("search.records",
		res.GetCollection(func(r res.CollectionRequest) {
			start := ParseInt(r.ParseQuery().Get("start"))

			records, err := PerformSearchQuery(start) // Query your database
			if err != nil {
				r.Error(err)
			} else {
				r.Collection(records)
			}
		}),
	)
	/* ... */
}

In the example above, I used string for the self/prev/next links. But now, the href-links are actually supported using soft references, a new feature that is will be coming in the upcoming release that is planned at the end of this week. :slight_smile:

Hope that helps.

/Samuel

Ps. Deeply nested static structures will later also be supported, but those won’t be updated in real-time.

#18

Excellent, thank you for that.

while the model wont really work well for what I was experimenting with it on until it can handle nested structures better. Now I’m curious about using it purely for cached data e.g. ephemeral messages, lets say I want to create a super simple chat, does the model have to actually return anything or can I use change events to populate the cache and have the calls to the model simply authorize and return whats in the cache? basically just pub/sub with a cache?

I’m excited for the next release. I have to say, I am truly amazed at what you have accomplished, I truly enjoy using resgate, thank you for all your hard work.

update I answered my own question about the cache, yes it works as long as there is an active subscriber to the topic, once the subscriber leaves the cache is cleared shortly after… which is great for a super simple chat that doesn’t store any conversations.

#19

is there a reason that the model object which has integer keys fails to work properly?

e.g

{
  1: {from: 'someone', to: 'someone else'},
  2: {from: 'another', to: 'yet another'}
}
#20

Glad that it is working well for most parts :grin:
I must admit, I am kind of curious as what you are using it for.

Yes. What you describe is possible. It works for simple stuff. Just be aware that, any restart of Resgate or restart of the service (due to system.reset), will clear the model in Resgate’s cache. Or, as you said, if noone is subscribing to the model.

A bit hacky but completely allowed :slight_smile: