Propagation and Pagination with Promises

GitHub API for querying users & repositories

In this article, we will look how we used the GitHub API for querying users and repositories of an organization. In particular, we will look at how promises were combined to make the data retrieval easier.

Querying GitHub is relatively easy with the GitHub API. The GitHub API is REST based and provides access to your Activity, Gists, Issues, Organizations, Pull Requests, Repositories, and Users, among other things. There are libraries available for most popular programming languages, abstracting you away from the underlying REST calls.

Like most REST APIs, you will need to perform several queries to extract the data you need. For example, from an Organization, you can access the Users, and from each User object, you can perform queries to get the users Name or EMail Address. From that information, you can then perform additional queries to get the users Role within the Organization.

GitHub also paginates the data that’s returned. So you may need to make several calls just to get the list of users. Handling all these REST based calls in a non-blocking language like JavaScript can be tricky. In the end, we just wanted a List of users, their names, email address and role, but how do you get there?

Combining promises

In JavaScript, the answer is to Use a Promise. A Promise object represents the eventual completion (or failure) of an asynchronous operation. But in this case, it wasn’t just one promise. We needed a Promise for each query, each page we loaded and each sub-query. So how did we structure this?

To accomplish this, we ended up creating a composition of promises. A top-most Promise that is composed of other Promises. The top-most promise settles once all the sub-promises are fulfilled or if one of them is rejected. We used the node-github library and ES6 promises.

One function to retrieve all data

function resolveAllUsers(res) {
    let promises = [];
    for (let user of res.data) {
        promises.push(resolveUser(user));
    }
    if (github.hasNextPage(res)) {
        promises.push(github.getNextPage(res).then((res) =>
            resolveAllUsers(res)
        ));
    }
    return Promise.all(promises);
}

function resolveUser(user) {
    return github.users.getById({ 'id': user.id }).then((githubUser) =>
        github.orgs.getOrgMembership({
            'org': '<your-org>',
            'username': user.login
        }).then((orgMember) =>
            `${user.login}, ${githubUser.data.email},` +
            `${githubUser.data.name}, ${githubUser.data.html_url},` +
            `${orgMember.data.role}`
            )
    );
}

The resolveAllUsers function loops through all the users and stores a Promise for each subsequent REST call. If the data is paginated (and more pages exist), then another composite Promise is added to the end of the list for those results. This will walk all the pages recursively until no more data is available.

Finally, the call to Promise.all(promises) indicates that this Promise should settle when all the sub-promises are fulfilled or one has been rejected.

const flatten = list => Array.isArray(list) ? list.reduce(
    (a, b) => a.concat(Array.isArray(b) ? flatten(b) : b), []
) : list;

function getMembers(organization, role) {
    return github.orgs.getMembers({
        'org': organization,
        'role': role
    }).then((res) => resolveAllUsers(res))
        .then((data) => flatten(data));
}

The resolveAllUsers function is invoked when the github.orgs.getMembers REST call is resolved (in the getMembers function), and this returns the top-level Promise. When this resolves, we know all the data is available.

Because of the pagination, the actual list of Promises is structured as follows:

[1, 2, .., 30, [ 31, 32, .., 60, [ 61, 62, .. 90 ] ] ]

which isn’t want we want. We solve this by flattening the final data before returning the result from getMembers.

Finally, the getMembers can be invoked using:

getMembers(settings.organization, 'all').then((data) => {
    for (let item of data) {
        console.log(item);
    }
    console.log(`Size: ${data.length}`);
}).catch((result) => console.log(result));

If the Promise returned by getMembers is fulfilled, then we know we have all the data we requested. If it was rejected, then we simply print the reason why.

Get scripts and analyze your organization

If you’re interested in more details or would like to use the GitHub scripts I wrote to analyze your own organization. I’ve made this work available on github. Enjoy!