Skip to content
This repository was archived by the owner on Aug 6, 2021. It is now read-only.

WIP: A more sophisticated set of blueprints #9

Merged
merged 18 commits into from
Mar 6, 2015
Merged

WIP: A more sophisticated set of blueprints #9

merged 18 commits into from
Mar 6, 2015

Conversation

mphasize
Copy link
Owner

Hej, I've been working on an "advanced" set of blueprints for some time now and I think it's time to discuss this.

As a preparation I enabled the generator to support different "flavors" of blueprints and I'm keeping the original blueprints around as a "basic" flavor and added the new blueprints as an "advanced" flavor.

So what do these new blueprints do?

  • enable full control on how the API REST handles associations/relationships: url links, indexing and sideloading supported.
  • add meta information to find queries, which enable pagination on the client side

What needs to be done?

  • remove the need to install lodash and pluralize as separate dependencies
  • update blueprints with latest fixes from Sails v0.11
  • add the camelCase fix for sideloaded association names in the JSON output
  • gather some feedback from the community: does this work/feel right? maybe some better naming for configuration options?
  • test how this works on Mongo/NoSQL databases
  • optimized performance of the index option on Many-to-Many relationships, i.e. on SQL we can just load the indexes from the join table without actually performing a full join.
  • simplify buildResponse to remove single record output in object style. Instead use array, as it is apparently preferred by Ember Data even for single records.

Happy to get some feedback on this and do some further planning...

@mphasize
Copy link
Owner Author

@Globegitter please take a look at this and let me know what you think? (Also... I need to add the camelCase fix here...)

@wayne-o
Copy link

wayne-o commented Feb 19, 2015

👍

@Globegitter
Copy link

@mphasize Good that you are starting a discussion here. After seeing json-api/json-api#341 and emberjs/data#1988 (comment) I was thinking maybe we can have a set of jsonapi blueprints and also rename it to that, to leave it more open to a wider audience that wants to use jsonapi.

I haven't looked at the latest specs properly yet and it is of course still WIP but I think it would be the most future-proof decision.

So what are your thoughts about that? Also /cc @wayne-o @kriswill @mgenev

But back to your original proposal, how exactly would the differentiation be between basic and advanced? Would that be two different sails generate commands? I think that is a great idea - what do you think about embedded support?

Will also take a look at the code itself now :)

@mgenev
Copy link

mgenev commented Feb 19, 2015

I like this idea a lot and I will gladly work on it. I was extending the current blueprints yesterday to support geo querying with mongo, postgres has that ability too with with postGIS.

I know that @kriswill has a good convention for a pod-like api structure which would be awesome.

@kriswill
Copy link

For some reference on JSON:API take a look at the Endpoints project and it's Example project. This implementation of the spec is still young, but is the most robust one so far.

@mphasize
Copy link
Owner Author

Wonderful! So in order:

  • JSON API: when I originally started implementing these blueprints I wanted to adapt the json-api standard, but soon had to realise that Ember was not (and still is not) a 100% compliant. My goal was to support Ember's DS.RESTAdapter out-of-the-box as much as possible, so these blueprints turned out quite Ember specific. My idea would be to evolve the blueprints along with the development of Ember Data, getting closer to the (eventually final?) json-api standard step by step.
  • Flavors: the differentiation between "basic" and "advanced" is essentially the amount of setup you need to do. For the sake of not breaking stuff, I didn't touch the basic blueprints at all, but I think that eventually (after double-checking) the advanced blueprints can be set to a default configuration that emulates the basic behaviour, so that an "upgrade" is possible without breaking existing code.
    The original idea of flavors goes a bit further though, because I would really love to support the "embedded" records api style that Ember can consume (with some setup) and I would also love to support a "pure" json-api flavor. If that warrants renaming the whole project or branching a separate generator for sails – no idea.
    On the practical side: at the moment it's simply another command line argument to pass to the generator.
  • Geo querying: nice! I've worked on some geovis projects in the past and I can get easily excited for this kind of stuff. @mgenev Please share what you got! :-)

@kriswill
Copy link

Also check out ember-data-endpoints which is being done by @bmac, core ember and ember-data contributor, and the original author of the ember-data-sails-adapter project.

@wayne-o
Copy link

wayne-o commented Feb 20, 2015

I am really interested in this thread. It all sounds great. I am going to
be busy with my son the next couple so wanted to chime in and make my
interest explicit! I'll be very happy to contribute towards this after the
weekend :)

On Thu, Feb 19, 2015 at 7:56 PM, Kris Williams notifications@github.com
wrote:

Also check out ember-data-endpoints
https://github.com/endpoints/ember-data-endpoints which is being done
by @bmac https://github.com/bmac, core ember and ember-data
contributor, and the original author of the ember-data-sails-adapter
https://github.com/bmac/ember-data-sails-adapter project.


Reply to this email directly or view it on GitHub
#9 (comment)
.

--------------------

w://

t: 07508 215 459

--------------------

@mphasize
Copy link
Owner Author

Some more info / questions around configuring json api output:

In the new blueprints I am introducing 2 ways to configure how associations are handled in the json output:

  • in the global sails configuration
  • on a model attribute

In myapp/config/models.js

module.exports.models = {
  associations: {
    list: "link",
    detail: "record"
  }
};

An in myapp/api/models/user.js

module.exports = {

  attributes: {
    name : "string",
    posts: {
      collection: "post",
      via: "user",
      includeIn: {
        list: "record",
        detail: "record"
      }
    }
  }
};

Depending on the blueprint, it will either use the list or detail value. The current mapping is

create ---> detail
find ---> list
findOne --> detail
update --> detail

What's your take on such fine-grained control and specifying this on the model ?
What do you think about the naming of list, detail and includeIn ... ?

@albertosouza
Copy link

@mphasize lets make a sails hook for this feature !
something like sails-hook-ember-blueprint
Sails hook has suport to updates with npm modules i can start the structure and setup it

@mphasize
Copy link
Owner Author

@albertosouza I think the scope of the "advanced" blueprints is a bit broader than what I can see in the other hooks you mentioned here https://github.com/mphasize/sails-ember-blueprints/issues/4
Instead of just parsing and changing the result of the final controller output, these blueprints aim to control json output depending on the actual request use-case (i.e. find a list, vs. get record details) and they also want to accept data from Ember (for create/update/etc.).

Or did I miss something?

@albertosouza
Copy link

@mphasize yes, you are right but we can create a hook to add these blueprints in the sails, i can help with the development of the hook as it will be useful for my projects

change sails-generate-ember-blueprints to sails-hook-ember-blueprints

Setting `res.status(201)` but then using `res.ok()` afterwards would reset the status code to 200, which we don’t want.
… dependencies and removes the necessity to install `lodash` and `pluralize` after installing the generator.
@mphasize
Copy link
Owner Author

@Globegitter Any more thoughts on this?

I just added the "shallow" blueprint generation we discussed earlier...

@Globegitter
Copy link

@mphasize Not quite sure about the different benefits of hook vs generators at this point (even though I am using hooks and liking them). It would not fill up the blueprints folder then?

And will look over the naming tomorrow if it makes intuitive sense to me :)

@Globegitter
Copy link

And yes 👍 👍 on tests :P Could potentially help on that front a bit.

} );
},

emberizeModelIdentity: function ( model ) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we are using lodash 3.x we don't need that helper function anymore and could just use https://lodash.com/docs#camelCase

@Globegitter
Copy link

@mphasize Sorry I just committed what should have been a PR. But it fixes everything that is mentioned in #8 and in the comment above.

Do you want me to revert, or is it fine like that? I do have this in my project currently and is working well.

@mphasize
Copy link
Owner Author

No worries. I will hold the next "publish" and version bump until we got it all figured out and we need the fix anyway. Is the whole camelCase issue resolved with that?

@Globegitter
Copy link

As far as I can tell yes. But also I haven't taken a step back yet and though it these are all the cases that could come up.

@wayne-o
Copy link

wayne-o commented Feb 25, 2015

I was already on it lol!

WORKS LIKE A CHARM!!!!

Good job guys - seriously!

On Wed, Feb 25, 2015 at 7:13 PM, Marcus notifications@github.com wrote:

@wayne-o https://github.com/wayne-o Go for it! There's lots of ideas
and discussion (and therefore changes to come), but the current code should
be working.


Reply to this email directly or view it on GitHub
#9 (comment)
.

--------------------

w://

t: 07508 215 459

--------------------

@mphasize
Copy link
Owner Author

@Globegitter I will need a bit more time to think through all the ideas and suggestions - hopefully towards the end of the week, but maybe I can answer this straight away:

How the sideload is performed:

  • in the find or findOne blueprint we check the associations defined on the model
  • in the advanced blueprint we furthermore check how these associations should be included
  • if an association is supposed to be indexed or embedded/sideloaded, we use an adapted version of the original populateEach, telling the Waterline find query to populate the associations (hence a minimum version of Sails 0.10 required)
  • after the find query comes back with all the associations embedded (since that is the original Waterline behavior) the buildResponse function (formerly known as emberizeJSON) gets to work on the data structure, analyses the model again for association declarations and pulls the records out into a sideloaded data structure. (It really is just shuffling objects around in a tree).
  • for associations that shall be indexed, the records get flattened into ID arrays (future optimization would be to not even load the records - just the indexes, see waterline#532)
  • for associations that shall only be linked to (we didn't populate the query) and now we generate a links object with the urls to load further data

Does this clarify what happens for you?

@mphasize
Copy link
Owner Author

@Globegitter Btw. at the point where we tell Waterline which associations to populate, we could look for further parameters in the API request to change what is included/sideloaded.

@mphasize
Copy link
Owner Author

@wayne-o Thanks and good to hear that it's working. :)

@mphasize mphasize mentioned this pull request Mar 1, 2015
@mphasize
Copy link
Owner Author

mphasize commented Mar 2, 2015

Alright... back to this again.
As we can see with #13 and my own ambitions regarding the accessControl hooks into models, there's a need to dive deeper into the relationship between request and model and what's supposed to happen.

However, it feels like we're overloading the scope of the blueprints when we introduce all of this, so I took a step back and had another look at Sails policies (because that's where stuff like access control) should belong according to the Sails philosophy. Digging deeper I found a couple of convenient things that Sails sets in req.options and that is accessible in the policies, so I am pretty confident that I can create a policy that handles the accessControl stuff and links to functions in the Model or wherever we want to put them. It should also allow us to keep the blueprints neat and clean. I'll do some further experiments today and if everything turns out well, I will probably separate the accessControl efforts into their own project.

* @param {[type]} associations [description]
* @param {Function} done [description]
*/
populateIndexes: function ( parentModel, ids, associations, done ) {
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can anybody verify that this method runs as expected on a Mongo/NoSQL database? (It's used in find and findOne blueprints). Set sideloading to index to enable this.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mgenev Maybe you can give that a try at some point?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should be able to give this a spin later today

On Mon, Mar 2, 2015 at 2:50 PM, Markus Padourek notifications@github.com
wrote:

In templates/advanced/api/blueprints/_util/actionUtil.js
#9 (comment)
:

  •       console.log( "Dev: Model " + model.identity + " is missing control " + action + "!" );
    
  •       if ( callback ) {
    
  •           callback( null, primaryArgument );
    
  •       }
    
  •   }
    
  •   return primaryArgument;
    
  • },
  • /**
  • * helper function to populate a record with an array for indexes for associated models, running various Waterline queries on the join tables if neccessary ( defined as: include -> index )
  • * @param {Waterine Collection} parentModel [description]
  • * @param {Array|Integer} ids [description]
  • * @param {[type]} associations [description]
  • * @param {Function} done [description]
  • */
  • populateIndexes: function ( parentModel, ids, associations, done ) {

@mgenev https://github.com/mgenev Maybe you can give that a try at some
point?


Reply to this email directly or view it on GitHub
https://github.com/mphasize/sails-generate-ember-blueprints/pull/9/files#r25602915
.

--------------------

w://

t: 07508 215 459

--------------------

@mphasize mphasize mentioned this pull request Mar 2, 2015
if(!_.isNumber(array[0]) && !_.isString(array[0])) { // this is probably an array of records
var model = sails.models[ pluralize(key, 1) ];
console.log("Sideloaded records model: ", model.identity);
Ember.linkAssociations(model, array);
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should fix #7

@mphasize
Copy link
Owner Author

mphasize commented Mar 2, 2015

Alright, this has been an enlightening discussion throughout for me and inspired a lot of good stuff, but this PR is getting a bit heavy and difficult to navigate, so...

I documented further todos in the roadmap issues #17 and #18. Please feel free to add more suggestions and continue discussion over there. Since this PR will not change existing setups, I'm going to merge it into master as soon as @mgenev or @wayne-o give us (hopefully) positive feedback on MongoDB. Then we can tackle the remaining issues in other PRs.

@mphasize mphasize mentioned this pull request Mar 2, 2015
7 tasks
@Globegitter
Copy link

@mphasize Yep that sounds like a good way forward :)

@mphasize
Copy link
Owner Author

mphasize commented Mar 5, 2015

Any news on Mongo?

@mgenev
Copy link

mgenev commented Mar 5, 2015

Sorry, no, I set everything up for a test on monday, but had to go to emberconf, now I'm back and I should be able to do the test finally.

@mgenev
Copy link

mgenev commented Mar 6, 2015

@mphasize i'm not seeing if i'm installing from the branch correctly. I downloaded this branch and put it in my new project's node_modules dir, I did generate advanced, and the crud methods now point to the /node_modules/sails-generate-ember-blueprints. However, the /api/blueprints/_util/actionUtil remained the old one and when i try to lift I get:

module.js:322
    throw err;
    ^
Error: Cannot find module 'lodash/array/uniq'
    at Function.Module._resolveFilename (module.js:320:15)
    at Function.Module._load (module.js:262:25)
    at Module.require (module.js:349:17)
    at require (module.js:368:17)
    at Object.<anonymous> (/Users/martin.genev/Projects/advanced-sails-ember-blueprint/api/blueprints/_util/actionUtil.js:5:21)
    at Module._compile (module.js:444:26)
    at Object.Module._extensions..js (module.js:462:10)
    at Module.load (module.js:339:32)
    at Function.Module._load (module.js:294:12)
    at Module.require (module.js:349:17)
    at require (module.js:368:17)
    at /Users/martin.genev/Projects/advanced-sails-ember-blueprint/node_modules/include-all/index.js:129:29
    at Array.forEach (native)
    at requireAll (/Users/martin.genev/Projects/advanced-sails-ember-blueprint/node_modules/include-all/index.js:44:9)
    at /Users/martin.genev/Projects/advanced-sails-ember-blueprint/node_modules/include-all/index.js:54:23
    at Array.forEach (native)

am i missing a step somewhere, doing something wrong?

_EDIT_
I copied the new action util manually and ran install lodash and util and got it to work.

@mgenev
Copy link

mgenev commented Mar 6, 2015

@mphasize If I understood you correctly you wanted to see if the array of index as an option works with mongo. To do this, I set this in the global model config:

associations: {
    list: "index",
    detail: "record"
  },

This did indeed with no visible error produce the array like this (vendors belong to users):

{  
   users:[  
      {  
         password:"$2a$10$2j4HYXFSJY3HIwlQVaISBuyjoT.Dn/uWp3lcYrr..VxJ/dmhnsX46",
         username:"email@gmail.com",
         firstName:"martin",
         lastName:"genev",
         userType:0,
         createdAt:"2015-02-23T03:14:06.941Z",
         updatedAt:"2015-02-23T03:14:06.941Z",
         id:"54ea9afe0ac0c5c35ca30244",
         vendors:[  
            "54ec0029ca7f8a94b15182d7",
            "54efc5bfc7db589a6e521af1",
            "54efc5edc7db589a6e521af2"
         ]
      },
      {  
         password:"$2a$10$pToIzzVEMI.SRBlRJb2SCODqLLWwbddVn1dkVe/nDN/udkkN7gDMG",
         username:"anotheremail@gmail.com",
         createdAt:"2015-02-27T05:02:51.636Z",
         updatedAt:"2015-02-27T05:02:51.636Z",
         id:"54effa7b4fc3b5892da2f041",
         vendors:[  

         ]
      }
   ],
   meta:{  
      total:2
   }
}

After that I switched to:

associations: {
    list: "record",
    detail: "record"
  },

And this once again with no error produced the entire records sideloaded:

{  
   users:[  
      {  
         vendors:[  
            "54ec0029ca7f8a94b15182d7",
            "54efc5bfc7db589a6e521af1",
            "54efc5edc7db589a6e521af2"
         ],
         password:"$2a$10$2j4HYXFSJY3HIwlQVaISBuyjoT.Dn/uWp3lcYrr..VxJ/dmhnsX46",
         username:"email@gmail.com",
         firstName:"martin",
         lastName:"genev",
         userType:0,
         createdAt:"2015-02-23T03:14:06.941Z",
         updatedAt:"2015-02-23T03:14:06.941Z",
         id:"54ea9afe0ac0c5c35ca30244"
      },
      {  
         vendors:[ ],
         password:"$2a$10$pToIzzVEMI.SRBlRJb2SCODqLLWwbddVn1dkVe/nDN/udkkN7gDMG",
         username:"another@gmail.com",
         createdAt:"2015-02-27T05:02:51.636Z",
         updatedAt:"2015-02-27T05:02:51.636Z",
         id:"54effa7b4fc3b5892da2f041"
      }
   ],
   vendors:[  
      {  
         name:"mhm",
         description:"981118314",
         urlSegment:null,
         address:"Seattle, WA",
         location:{  
            type:"Point",
            coordinates:[  
               -122.3320708,
               47.6062095
            ]
         },
         user:"54ea9afe0ac0c5c35ca30244",
         createdAt:"2015-02-24T04:38:01.790Z",
         updatedAt:"2015-02-24T04:38:01.790Z",
         id:"54ec0029ca7f8a94b15182d7"
      },
    ...
    /// rest of vendors records trunc'd for paste ///
   ],
   meta:{  
      total:2
   }
}

As you can see above I'm also getting meta total. I tested 'link' as well and it produced:

links: {
    vendors: "/api/v1/users/54ea9afe0ac0c5c35ca30244/vendors"
}

which I then hit in the browser and it gave me the list of the vendors for that user with no hesitation.

Applause! This is great work and I'll be including it in app as soon as you merge it and give it a thorough test over time in different situations.

My question is, and excuse me if the docs answer it but i haven't noticed, do I have a way of altering this behavior from Ember, on a per request basis like this:

this.find('users', {assocations: {list: 'index'}}

Or whether this clicks with the ember data's 'async' flag on associations I set on the client. This way I could finely control my app's loading behaviors and load whole records async later if I need to as each situation calls for.

Once again, this is awesome!

@mphasize
Copy link
Owner Author

mphasize commented Mar 6, 2015

@mgenev Wonderful news! Thanks a lot for trying it out. The results look the way they should be, so I assume that there's at least no general problem and we will have to iron out bugs when they manifest.

Concerning your question: At the moment you cannot control which associations will be included on a per-request basis. It's on the roadmap, but it's not in the blueprints yet. There's a bit more security thinking necessary, as people requesting data from your API can easily change the query and potentially create a situation with a lot of heavy includes.
However, at the moment you can control behaviour at a per-property basis on each model individually using the includeIn keyword.

Example:

// in /api/models/user.js
module.exports = {
    attributes: {
                // ...
        posts: {
            collection: "post",
            via: "user",
            includeIn: {
                list: "link",
                detail: "link"
            }
        }
    }
}

Thanks again for your feedback! I'll go ahead and merge this into master now, then I'll give it a bit more time to stew before I release it to npm.

mphasize added a commit that referenced this pull request Mar 6, 2015
A more sophisticated set of blueprints, install using `sails generate ember-blueprints advanced`
@mphasize mphasize merged commit 2d5b4a4 into master Mar 6, 2015
@mgenev
Copy link

mgenev commented Mar 6, 2015

Excellent. Please give me heads up when it's on npm and I'll add it to https://github.com/mgenev/how-to-sane which would allow a whole bunch of people to report bugs. I'll be away from computers for 2 weeks starting tonight, but i'll try to keep up with reading on my tablet

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy