With the documentation changes wrapped up, but holding off on PR’s until things solidify up a bit more from the code scrub process, it was time to move on to the OpenID deep dive and review. Up until now I’ve been working with an authorization workflow that required me to request a new token ever 24 hours and for the user to authenticate it. I wasn’t sure how much of that was because of the flow I chose or intrinsic to how it was coded up. As I continued to go over the OpenID documentation and other articles on the process I just couldn’t get it working. It was then clear to me that what I needed was an example to help me.
Luckily Nov Matake created some example projects to go along with his OpenID gems, one for the OpenID Connect Provider (the server side) and one for the OpenID Relying Party (the app side). I figured with that everything would be good to go. After all this was the same code he had running up on Heroku but I wanted to see the nitty gritty details and set it up on both sides since I was going to need to do that with Diaspora and the test harness, or any other API use case I may be interested in. As I had come to find out quickly these projects have never been updated. They still rely on old versions of Ruby and Rails. Instead of trying to downshift everything to these versions I decided to fork the projects and get them running under Ruby 2.4+ and Rails 5. Unfortunately that derailed my entire Diaspora development effort for the day. The upside is that the community will have modern versions of these projects to use. I intend to polish them up a little more and then issue a PR back to the original project. My versions however can be found on my GitHub profile with the Connection Provider here and the Relying Party here.
In the process of doing these upgrades I was able to learn a lot more about porting Ruby code up from older versions. I also got a much better understanding of some OpenID flows. I’m going to use that to continue to move forward on the review of the implementation in the API and looking at client side implementation details. Because of the complexity of that whole process I think that’s probably something developers can use a good amount of help for via blog posts and examples.
- Documentation updates are complete but waiting for PRs for after the code scrub
- Updated Ruby on Rails OpenID examples from Nov Matake to work under Rails 5
Follow the discussion on Diaspora (or this link if you don’t have a Diaspora account with the web+diaspora protocol enabled in your browser).
You can follow the status dashboard at this Google Sheet as well.
Coming up with a paging infrastructure for the API while looking at all of the ways it could be used and abused hasn’t been fun. Not that it hasn’t been totally worthwhile. I’ve actually learned a lot more about some of the nuances of how ActiveRecord and related libraries are building up their queries. I’ve thought a lot more about the nature of the queries within Diaspora too. At the same time my head is numb and for all of the effort I only got a half completed design and less than 100 lines of code across two classes, not that more lines is necessarily better.
So what we will have are two paginator types: index based and time based. The standard methods across the two are:
- page_data: returns the current page of data for passed in query
- next_page: returns information to go to the next page of data
- previous_page: returns information to go to the previous page of data
The previous/next page functions will either return a new paginator object that corresponds to the next page or it will return a string that represents query parameters that can be passed back out from a REST endpoint.
Both paginator types take a query object that will then have additional paging stuff wrapped around it. If one is doing an index-based query this is just wrapping the WillPaginate library. However if one is doing a time based query then it’s a little more complicated than that. We aren’t simply moving around indexes we actually are doing some time math. That is all coded directly in the class. The big difference between the two comes in how the ordering happens on the SQL query. In the case of both you can pass in an ordered query without throwing an error. However in the case of the IndexPaginator one probably wants to pass in their preferred order otherwise they’ll get whatever the natural order from the database is. In the case of the TimePaginator it wants to keep control over sorting by whichever time field the calling code is using. Therefore adding an additional sort could create confusing results.
Now that the paginators are done I need to add a present class that knows how to turn the query parameters into a “link” field with full URLs, per the API specification, and to update the services to call into and return the paginated data instead of their current form. I think I’ll do one that uses indexes, like contacts, followed by one that uses time, like user posts, and then start filling it out the rest of the way from there.
- Completed playing around with the base pagination classes and completed them.
- Starting to wire in first pagination into some first endpoint
The last couple of days has been a lot of heavy effort of slogging through some ever increasingly complex changes to get the API going. I started with what I thought was going to have a relatively easy time with the notifications however the deeper I went into the more I realized that I either had to come up with some relatively (for me anyway) complex queries to populate some of the return types or I have to settle for some N+1 type query behaviors. “N+1 queries” are one where you pull the results one piece at a time. That’s fine for smaller data sets, like five or ten or something, but if you are dealing with hundreds of entries you are really thrashing your system. So I got about half way through the notifications API and then put it on the shelf and moved on to the API was dreading the most: Photos.
I was really psyching myself out about having to deal with the whole image file upload part of the Photos API and then the subsequent tie in with the Posts API. It shouldn’t be that complicated but these are things I had never done in Rails or with the Kotlin Fuel framework. How would they interact? How difficult would the security checks be? You get the idea. It did take several hours of figuring out what the current controller is doing and then how I wanted to refactor the more complicated operations into a service but I got there. Once I had that I had to test the whole aspect of limited posts et cetera, which I hadn’t done as well as I had thought previously. Thankfully my Ruby unit tests were solid I just had some hiccups in my test harness.
At the end of the day we have the Photos API and the Posts API working with the photos perfectly, to the point where I was able to make a fully populated post including with an image that was uploaded externally as well. That means I’m going to jump back on the Notifications API to wrap that up and all that’s left is the Search API.
- Partial Progress on the Notifications API but shelved to figure out queries later
- Posts API is feature compleet with full tests
- Was able to create an entirely populated post with the respective images from scratch using an external application for the first time ever in Diaspora (see this post)
- 1.5 Endpoints left to go to be feature complete
After slogging away for most of today on the Photos API, with lots of needing to understand how things work and a couple more tweaks before it was ready, I decided to celebrate by showing the ultimate progress report: a screenshot. What is so special about this screenshot? It is the first post in Diaspora that has been fully made by an external application. The “external application” in this case is a test harness written in Kotlin which is designed around the API spec. This test harness first uploaded the image file, then it created the post with every feature a post can have including: location, polls, and references to other users. The post was written by a “user3” (for testing might as well stick to simple names). This is a screenshot from user1’s perspective. Notice that they also got the expected notification. Yes it’s still a bit of a ways from done but it’s still a great milestone, so I’d say it’s time to celebrate for a bit before getting back to it :).
Brief update from today on the Diaspora API development progress:
- On the Users API turns out we probably still want to have the contacts endpoint if only for the primary user since the Contacts API works on a per-aspect level the way it is mapped. Whether that method shows up in Contacts API at a different mapping or on the User itself is still TBD but it will be a change to the spec.
- The Post Interactions API is feature complete with full tests and the completed test harness.
- Work has begun on the Notifications API. This is the first change I’ve done that will require a DB migration, adding a new GUID column to notifications, so this is going to take a bit longer for me to complete as I do background research on that.
At this point it’s actually easier to look at what is left to do versus what we have done (which is a huge plus sign):
- The only two endpoints that haven’t been touched are Photos and Search. Once these are done (along with work on Notifications) the entire API spec will have been implemented.
- Implement a new poll interaction method for answering a poll through the API
- We need to implement paging on several of the endpoints. This technique will be similar to how it’s done in the core controllers but it has to be different because the return type needs to have the next/previous pages and the corresponding format needs to honor that. The actual mechanics of the queries are pretty much the same though so grafting them into the existing feature complete controllers should be relatively easy.
- Right now the OpenID integration works well enough for testing but it currently requires revalidating the app every 24 hours. This has to be tweaked to be more reasonable. There may be some refactoring in there as well.
- The Posts API Endpoint accepts any photos currently, including those that are already attached to another post. This is not consistent behavior and has to be corrected to only allow a “pending” photo to be added.
- Sweep of all of the APIs for consistency on security, service initialization (where appropriate), params parsing idioms, etc.
- Sweep through the unit tests to make sure that edge cases are covered in the same way
- Documentation updates to account for things discovered during the development (error codes added, format tweaks etc.)
It’s been two weeks since my last Diaspora API Dev Progress report but that’s not because nothing has been going on. Between the RubyConf 2018 attendance last week and this week being a holiday week there was definitely a drop off in how much development time I put into Diaspora, and therefore mostly into the API. However over that time there has been some development progress:
- All of the previous work has been successfully merged down into the main API branch.
- The Contacts API is feature complete with full tests and the completed test harness
- The Users API is feature complete with full tests and test harness with the exception of the User Contacts API method. That method was supposed to be able to return another user’s contacts if that user allowed that. However that feature no longer exists in Diaspora so I believe it is extraneous. If that’s agreed upon then this is feature complete and ready to go.
This week I should be able to apply a lot more development effort than I have been able to the past couple of weeks. Hopefully that translates into forward progress on some more endpoints. The trend seems to be that they are getting more difficult to knock out so my velocity is slowing. I guess it’s better than being stymied in the beginning.
Yesterday was the first day in several I could commit to real time towards D* again. After getting back up to speed and making the status post I went on into the API development again. I was able to make some good progress on some brand new endpoints. The first one I worked, which is the first that needed from scratch coding of the main code, was the Tag Followings controller. The day before I had struggled getting Rails to make the POST for creating tags work against the spec. However after talking it over and thinking about it it was the spec that needed changing. In another software framework I could just make it work but relying on the auto-wiring in Rails brought the design flaw nature to light. With a simple change starting yesterday real development of the Tag Followings endpoint started.
The methodology I’m using when developing the new controllers is as follows. First, I want to get the basic infrastructure in place and the tests. That means that the first phase is to write the skeleton of the controller code, the skeleton of the RSpec tests, and to wire the two together. I make sure that the routes behave the way I think they should according to the API Spec without worrying about returns etc. The skeleton of the controller should implement all routes. The skeleton of the unit tests should be testing for happy path and reasonable error conditions. So that’s stuff like: the user passes the wrong ID for a post that they are trying to comment on, or an empty new tag to follow, etc. I then go over to the external test application and code up the corresponding code in there as well. With everything running I make sure that the endpoint is reachable from the outside (which it should be), but don’t worry about returns, processing etc. If it’s possible to setup fake returns easily I do that otherwise I just ensure the proper methods are called. After all of that is coded and committed then it is off to filling in the controller method by method. For each one coded up I complete the unit tests and the external test harness interactions as well. Once that’s all done then I move on to the next one. In some cases, like Tag Followings, there needs to be refactoring elsewhere which has implications on the above flow. I usually do those pieces before coding the controller. It is at the design time that whether I should be using common code with another controller which may not exist as a Service component becomes apparent. If I need to make any changes over in other code I check that there are unit tests which properly cover the changes I am going to make, at least as best as I can tell, write those and then make the changes. This should minimize the possibility of disruption.
When interacting with Frank R. on the merge requests one of the pieces of feedback I got was that with everything compressed down to one commit it was hard to tell why I did certain things. As I code all of that is there but I’ve been rebasing everything down to one commit per endpoint so that when it comes time to merge the API branch into the main develop the log will look something like: Post API endpoint complete, Comments API endpoint complete, etc. To get around this I’m trying a new flow. When I think something is ready to be merged i’m doing a Work in Progress (WIP) Pull Request (PR). That PR has the raw commit history and the name “WIP” in the leader of the label. After a review and a thumbs up I’m going to rebase it down to one commit and then submit the final one for integration. By the time WIP is done the code is feature complete however and should be ready to be merged. I’m therefore counting WIP PR’s as the threshold for saying something is feature complete.
With all that said the three new endpoints that were feature complete as of yesterday are: Tag Followings, Aspects, and Reshares.
After a week of distractions I finally have a new update on the progress. We’ve successfully merged all the work done to date into the one main API branch and are now working on new features moving forward. The first feature we have completed with full tests and test harness interaction is the ability to manage and work with the user’s followed tags. So we have the full post lifecycle from before, and now tags done but not merged into the main branch yet.
The merging of the various side branches into the main branch is coming along. Because this isn’t being done as a primary job there is a bit of an expected delay between the pull request (PR) being generated and the branch being merged in. This is giving me the opportunity to work on other features on Diaspora though. The process is going along much faster than I expected it to, which is good. At this point we have merged the Likes, Comments, and Post Endpoints together. The PR on the Post Endpoint is now queued up however all of those changes exist in one branch. What that means is that I was able to perform a full Post life cycle test using the test harness. This means that we have an external application talking through the API and doing the following for a user:
- Creating a post
- Querying for the post and printing out it’s data
- Adding a comment to the post
- Liking to the post
- Printing out the comments and who liked the post
- Deleting their comment on a post
- Unliking a post
- Deleting a post
This is a very important step. Follow additional progress on the API Progress Google Sheet.
It’s been a few days since I’ve been able to put some real time into Diaspora development but I’m back today. Being back home from travel too means I can finally get past the blockers on the other branches. I’ve actually gotten all of the branches I had been developing on to feature complete status, with full tests, and the test harness fully coded against it. That means that through the API one can complete the entire Post, Comment, Like, etc. lifecycle for posts with all data types (regular, Photos, Polls, location, etc). Conversations are also feature complete with full test harness as well. Streams are also complete, however I haven’t tested with sufficient post volumes to test paging behavior. Now it’s going to be the trick of getting past the tech debt of getting them merged together into the API branch. Hopefully that’ll come in the next day or two. I’m going to spend some time doing other Diaspora stuff besides that as I work through those pieces as well. As always follow the progress on the API Progress Google Sheet. After the merge I’ll be moving on to the Tags Endpoint, the first endpoint that is a full from scratch development for me.
- Fully feature complete endpoints with full external test harness interaction completed are: Comments, Conversations, Likes, Posts, and Streams (except for paging behavior).
- Ready for merging of the side branches into the main API branch