Category Archives: Software Engineering

Diaspora API Dev Progress Report 25

With the documentation changes wrapped up, but holding off on PR’s until things solidify up a bit more from the code scrub process, it was time to move on to the OpenID deep dive and review.  Up until now I’ve been working with an authorization workflow that required me to request a new token ever 24 hours and for the user to authenticate it.  I wasn’t sure how much of that was because of the flow I chose or intrinsic to how it was coded up.  As I continued to go over the OpenID documentation and other articles on the process I just couldn’t get it working.  It was then clear to me that what I needed was an example to help me.

Luckily Nov Matake created some example projects to go along with his OpenID gems, one for the OpenID Connect Provider (the server side) and one for the OpenID Relying Party (the app side).  I figured with that everything would be good to go.  After all this was the same code he had running up on Heroku but I wanted to see the nitty gritty details and set it up on both sides since I was going to need to do that with Diaspora and the test harness, or any other API use case I may be interested in.  As I had come to find out quickly these projects have never been updated.  They still rely on old versions of Ruby and Rails.  Instead of trying to downshift everything to these versions I decided to fork the projects and get them running under Ruby 2.4+ and Rails 5.  Unfortunately that derailed my entire Diaspora development effort for the day. The upside is that the community will have modern versions of these projects to use.  I intend to polish them up a little more and then issue a PR back to the original project.  My versions however can be found on my GitHub profile with the Connection Provider here and the Relying Party here.

In the process of doing these upgrades I was able to learn a lot more about porting Ruby code up from older versions.  I also got a much better understanding of some OpenID flows.  I’m going to use that to continue to move forward on the review of the implementation in the API and looking at client side implementation details.  Because of the complexity of that whole process I think that’s probably something developers can use a good amount of help for via blog posts and examples.

In Summary:

  • Documentation updates are complete but waiting for PRs for after the code scrub
  • Updated Ruby on Rails OpenID examples from Nov Matake to work under Rails 5

Follow the discussion on Diaspora (or this link if you don’t have a Diaspora account with the web+diaspora protocol enabled in your browser).

You can follow the status dashboard at this Google Sheet as well.

Diaspora API Dev Progress Report 24

Yesterday I said the paging API was complete but needed to be reviewed.  The more I talked over some elements with people and in exchanges on Diaspora I realized there were a couple of tweaks I needed to do.  The first suggestion I implemented was to have paging on any endpoint that returns multiple elements.  The second thing was to have a parameter for specifying the number of elements requested.  I was pleased that supporting that feature was really just two lines of code to change.  However while in there I decided to beef up some other defense programming techniques in some other places.

After that was done I moved on to implementing the ability to vote on polls.  There was no home for it but since it is interacting with a post I put it on the Posts Interactions endpoint rather than create a dedicated endpoint with just one method.  It aliases to a path in the same way as the rest of the interactions as well so I think it’s consistent.  That also required a little moving things around from the existing endpoint into a service and then having both calling that.  Since there were no tests around that capability I ended up writing those as well.  With that done it’s time to move on to the documentation and then start hitting up the OpenID review.

In Summary:

  • Incorporated suggestions in the paging in the API
  • Completed the Poll Voting method
  • Moving on to documentation updates

Follow the discussion on Diaspora (or this link if you don’t have a Diaspora account with the web+diaspora protocol enabled in your browser).

You can follow the status dashboard at this Google Sheet as well.

Diaspora API Dev Progress Report 23

After a day of coding the paging is now in every endpoint that should have it.  That means that we have paging right now for:

  • Contacts
  • Photos
  • Posts
  • Comments
  • Notifications
  • Conversations (but not messages in conversations)
  • Search
  • Streams

Because of the size of the code changes I would imagine there will at least be some tweaking and I could imagine there being some larger refactoring afterward too but it’s in a solid, working, and as performant space as the existing standard endpoints so I’m happy with it.

Now it’s on to the rest of the checklist.  With the scopes being rounded out I’m going to hold off on the security review for a little while longer.  The first low hanging fruit I’m working on is adding to the API Spec the ability to vote on polls.  It was an oversight in the original design but it should be easy to do.  I just need to decide which endpoint to add it to.  After that I’m going to double back to the mundane documentation update task.  At that point I think it’ll be time to go through and get up to my elbows into the OpenID code and get ready to make changes for the new scopes.

In Summary:

  • Paging is now complete and ready for review
  • Starting work on voting on polls through the API

Follow the discussion on Diaspora (or this link if you don’t have a Diaspora account with the web+diaspora protocol enabled in your browser).

You can follow the status dashboard at this Google Sheet as well.

Diaspora API Dev Progress Report 22

Paging paging and more paging.  I haven’t been committing as much time to development the last few days as I’d like.  Some of that is frustration with the development process on the paging, which has been a lot of trial and error.  Some of it is just how my schedule is working out too.  There is progress there though.  I have what I’d consider to be the rounded out API Paging infrastructure in place.  It has migrated a bit since the last update since as I tried to use it I wasn’t happy with it.  I’m still not happy with it but it is suitable.   There will probably be some additional tweaking before final integration but what it allows is for us to have paging.  I ended up wringing out design problems by wiring it into the Aspects Contacts endpoint method (to test index-based paging) and the User’s Posts endpoint (to test time-based paging).  With all of that working and unit tests I’m now moving on to adding it to the rest of the endpoints. There has also been some additional discussions on the permissions scopes for the endpoint as well, and I think we’ve converged on a good final set.

In Summary:

  • Paging API infrastructure modified to current MVP (I think) status
  • Paging API now used in the Aspects Contacts and the Users Posts method
  • Rounding out finishing the endpoints and updating the test harness

Follow the discussion on Diaspora (or this link if you don’t have a Diaspora account with the web+diaspora protocol enabled in your browser).

Diaspora API Dev Progress Report 21

Coming up with a paging infrastructure for the API while looking at all of the ways it could be used and abused hasn’t been fun.  Not that it hasn’t been totally worthwhile.  I’ve actually learned a lot more about some of the nuances of how ActiveRecord and related libraries are building up their queries. I’ve thought a lot more about the nature of the queries within Diaspora too.  At the same time my head is numb and for all of the effort I only got a half completed design and less than 100 lines of code across two classes, not that more lines is necessarily better.

So what we will have are two paginator types: index based and time based. The standard methods across the two are:

  • page_data: returns the current page of data for passed in query
  • next_page: returns information to go to the next page of data
  • previous_page: returns information to go to the previous page of data

The previous/next page functions will either return a new paginator object that corresponds to the next page or it will return a string that represents query parameters that can be passed back out from a REST endpoint.

Both paginator types take a query object that will then have additional paging stuff wrapped around it.  If one is doing an index-based query this is just wrapping the WillPaginate library.  However if one is doing a time based query then it’s a little more complicated than that.  We aren’t simply moving around indexes we actually are doing some time math.  That is all coded directly in the class.  The big difference between the two comes in how the ordering happens on the SQL query.  In the case of both you can pass in an ordered query without throwing an error.  However in the case of the IndexPaginator one probably wants to pass in their preferred order otherwise they’ll get whatever the natural order from the database is.  In the case of the TimePaginator it wants to keep control over sorting by whichever time field the calling code is using.  Therefore adding an additional sort could create confusing results.

Now that the paginators are done I need to add a present class that knows how to turn the query parameters into a “link” field with full URLs, per the API specification, and to update the services to call into and return the paginated data instead of their current form.  I think I’ll do one that uses indexes, like contacts, followed by one that uses time, like user posts, and then start filling it out the rest of the way from there.

In summary:

  • Completed playing around with the base pagination classes and completed them.
  • Starting to wire in first pagination into some first endpoint

Diaspora API Dev Progress Report 20

Now that we’ve hit feature complete status it’s about getting more of the legwork down to get us really ready for integration.  The first necessary feature we need before that is paging.  As I wrote earlier, some endpoints don’t need paging and all of them technically have it as an optional thing.  However to be really useful we need to have paging for several endpoints like posts, photos, conversations, et cetera.  It looks like we can leverage a lot of the way we do paging in the lower levels for streams and just create a standard pager class that the API endpoints that need it can use.  I’ve laid out how I want to approach that so now it’s on to implementation.

Along with the progress on the paging there has been progress on other mundane areas.  All of these features were developed in side branches which needed to be reviewed and integrated into the main API branch.  We are down to one endpoint left before the API branch itself is feature complete, not just having the code.  All of the branches are orthogonal except for the routes.rb file and the en.yml messages file so it’s pretty easy integration but needs to be done properly.  In the mean time we are also having discussions about the finer grained permission sets that apps will request and users will be notified about.  So for example, an app could be given permissions to only read posts but read/write comments on posts, and so on.  The endpoints already check for read/write tokens but they are broad tokens.  Part of the next steps will be putting in the proper requests and making sure that the information presented to users is clear.

In summary:

  • All but one endpoint is integrated back into the API main branch
  • Started work on the API Paging infrastructure
  • Looking at the finer grained permissions for each endpoint

Open Source Dot-Net is 4 years old and going strong!

It seems like just a couple of years ago that Microsoft, the evil empire of the 1990s and early 200s, embraced open source and put the .NET ecosystem into the open source.  It was a shocking event which was meant with some pessimism by a community that had been bitten far too many times by the old mantra “embrace, extend, extinguish” from Microsoft (not that they were unique in this mantra).  It’s shocking that we are four years into this process but more shockingly is how well the .NET community is functioning.  This is not an “in the open source” which is code for “you can see the code but we are the developers.”  Microsoft, against all my expectations, has successfully built an open source community around open source .NET.  Take a look at the pull request statistics.  There is a substantial community element in most of the pieces (Chart and to read more it check out Matt Warren’s blog post on this):

If you look at the time series data he Warren has created it looks even more promising.  That’s not to say all is well for everyone in the .NET open source world.

As a person that tried to get back into it, to the point of polishing off SharpenNG to make it work in a post Java 7 world, I have to say that even with the improvements over the last few years the non-Windows platforms are still not first class citizens.  Development for .NET sings under Visual Studio, which of course only runs on Windows.  The old Xamarin Studio rebranded as Visual Studio Mac does provide a decent experience but still nothing in comparison.  People on Linux on the other hand are out in the cold.  Yes there are the command line tools and Visual Studio Code.  That works a lot better than I expected but you can feel how clunky that development is in comparison, and MonoDevelop seems to get worse and worse as time goes on.  When I think about dabbling with .NET again I think about trying Rider by JetBrains the next time.  Perhaps they’ve cracked the nut.  One thing I refuse to do is jump to Windows.

Related to all of that is the other elephant in the room: Microsoft doesn’t support UI development nor has any plans to on Linux.  There are open source alternatives like Avalonia and Eto.NET.  I know that Michael Dominic’s development shop was able to turn out a live geospatial cross platform app, Gryphon, using Avalonia so there can be some serious work done with this.  Maybe because of that official blessing from Microsoft isn’t needed, especially if Rider combined with the above fits the bill.  Maybe that’s the community evolving beyond Microsoft too?  Still, at this stage there is a second (or in the case of Linux third) class citizenship feel about it.  It’s orders of magnitude further along than I thought they would get though, which is a promising sign.

Diaspora API Dev Progress Report 19

We’ve finally reached the milestone we’ve all been waiting for.  With the completion of the Search API Endpoint the Diaspora API is now feature complete.  That doesn’t mean that it’s ready for integration into the mainline branch.  It also doesn’t mean that there isn’t more fundamental work that has to be done before it can be used on a production system.  It does however mean that we can start working on rounding out some of the other fundamentals and make our way in that direction.

The first thing that I am going to work on is the paging aspect to the API.  The API spec discusses paging as a thing that endpoints may or may not do.  Right now there is no paging.  That’s fine for some things, like getting a list of Aspects for a user.  It is a requirement for something like getting a list of a user’s posts or for getting your stream.  For non-developers who are reading this think of this as the piece that makes your “infinite scroll” work.  Diaspora has implemented this in other areas but it will have to work a bit differently for the API.  We’ve already had discussions about how we want it to work and there is a format specification for reporting it back.  It therefore should be relatively straight forward to get it implemented.  That is what I’m working on right now.  After that we’ll want to go over all of the new code with a fine tooth comb for style and idiom consistencies (beyond the automatic style checker), security reviews, etc.  Lastly we’ll want to get the OpenID authentication/authorization/etc. stuff polished up a bit.  Currently the app has to be re-registered every day.  That’s not going to be viable for a real user even if it is for testing.

Still, the fact we’ve reached a feature complete milestone is great news and I’m excited to be ending the weekend on that high note.

In summary:

  • Diaspora API is now feature complete
  • Search API endpoint, unit tests, and test harness are complete
  • User contacts endpoint implemented completing that endpoint
  • Beginning work on paging infrastructure for API endpoints that need it

To follow along with status please see the Google Sheet Dashboard.

Diaspora API Dev Progress Report 17

The last couple of days has been a lot of heavy effort of slogging through some ever increasingly complex changes to get the API going.  I started with what I thought was going to have a relatively easy time with the notifications however the deeper I went into the more I realized that I either had to come up with some relatively (for me anyway) complex queries to populate some of the return types or I have to settle for some N+1 type query behaviors.  “N+1 queries” are one where you pull the results one piece at a time.  That’s fine for smaller data sets, like five or ten or something, but if you are dealing with hundreds of entries you are really thrashing your system.   So I got about half way through the notifications API and then put it on the shelf and moved on to the API was dreading the most: Photos.

I was really psyching myself out about having to deal with the whole image file upload part of the Photos API and then the subsequent tie in with the Posts API.  It shouldn’t be that complicated but these are things I had never done in Rails or with the Kotlin Fuel framework.  How would they interact?  How difficult would the security checks be?  You get the idea.  It did take several hours of figuring out what the current controller is doing and then how I wanted to refactor the more complicated operations into a service but I got there.  Once I had that I had to test the whole aspect of limited posts et cetera, which I hadn’t done as well as I had thought previously.  Thankfully my Ruby unit tests were solid I just had some hiccups in my test harness.

At the end of the day we have the Photos API and the Posts API working with the photos perfectly, to the point where I was able to make a fully populated post including with an image that was uploaded externally as well.  That means I’m going to jump back on the Notifications API to wrap that up and all that’s left is the Search API.

In summary:

  • Partial Progress on the Notifications API but shelved to figure out queries later
  • Posts API is feature compleet with full tests
  • Was able to create an entirely populated post with the respective images from scratch using an external application for the first time ever in Diaspora (see this post)
  • 1.5 Endpoints left to go to be feature complete