Describing the foundations of the new PoP

We are re-architecting PoP to make it much simpler to code with it, faster, and more versatile, planning to release it around end of 2018 or beginning of 2019. The new PoP will feature an architecture based on the following foundations:

  • Everything is a module
  • The module is its own API
  • Reactivity

At the core of these changes is an upgraded API model, which will feature server-based components. Following the example set by GraphQL, which owes its success in part to being a specification instead of an implementation, PoP will be composed of the following layers:

  1. The API (JSON response) specification
  2. PoP Server, to serve content based on the API specification
  3. PoP.js, to consume the content in the client

Being built around an open specification, we can expect different implementations to emerge, for different technologies. The current PoP site will immediately provide the first implementation of PoP Server and PoP.js:

  • PoP Server for WordPress, based on PoP Server for PHP
  • PoP.js through vanilla JS and Handlebars templates

The API (JSON response) specification

In PoP, a component is called a module. Each module is either an atomic functionality, or a composition of other modules. Hence, starting from the lowest-level modules (the atoms), modules will be iteratively composed together until obtaining the highest-level module (the page). The new API model respects the structure of how modules are composed, and print all information needed at each module level.

In PoP, the logic of the application does not reside on the client-side, but on the server-side. The server indicates what javascript functions will be executed, on what elements on the DOM, and with what values. This will extend to reactivity: it is the server who will decide when to make a component be re-rendered. Reactivity is supported by splitting the information for each module into two groups: static, which is non-reactive, and stateful, which is reactive; the application in the server knows if to apply reactivity based on the querystate parameters added to the URL.

The API structure is divided into module configuration, database data, and additional meta entries. Once finished, it will look something like this:

modulesettings:
    static:
        foreach module {
            configuration
                module
                template
            jsmethods
            dbkeys
            datafields
            foreach nestedmodules {
                repeat ...
            }
        }
    stateful:
        foreach module {
            configuration
            foreach nestedmodules {
                repeat ...
            }
        }
moduledata:
    static:
        foreach module {
            feedback
            foreach nestedmodules {
                repeat ...
            }
        }
    stateful:
        foreach module {
            feedback
            foreach nestedmodules {
                repeat ...
            }
        }
datasetmoduledata:
    static:
        foreach module {
            dbobjectids
            foreach nestedmodules {
                repeat ...
            }
        }
    stateful:
        foreach module {
            dbobjectids
            foreach nestedmodules {
                repeat ...
            }
        }
datasetmodulemeta:
    static:
        foreach module {
            dataloadsource
            lazyload
            foreach nestedmodules {
                repeat ...
            }
        }
    stateful:
        foreach module {
            dataloadsource
            lazyload
            querystate
            queryparams
            queryresult
            foreach nestedmodules {
                repeat ...
            }
        }
modulejssettings:
    static:
        foreach module {
            configuration {
                foreach jsmethod: {...}
            }
            foreach nestedmodules {
                repeat ...
            }
        }
    stateful: 
        foreach module {
            configuration {
                foreach jsmethod: {...}
            }
            foreach nestedmodules {
                repeat ...
            }
        }
modulejsdata:
    static:
        foreach module {
            inputs {
                foreach jsmethod: {...}
            }
            foreach nestedmodules {
                repeat ...
            }
        }
    stateful:
        foreach module {
            inputs {
                foreach jsmethod: {...}
            }
            foreach nestedmodules {
                repeat ...
            }
        }
databases
    primary {
        ...
    }
    userstate {
        ...
    }
requestmeta
    entry-module
sessionmeta
    loggedinuserdata
sitemeta
    sitetitle

Let’s analyze the features provided by the API’s structure:

Concerning modules

The “Everything is a module” maxim implies at there will only be 1 module at the top of the page, from which we start composing all the other modules. This module is defined under entry-module. Then, rendering the page will simply be rendering the entry-module:

Concerning data

Not all modules fetch data from the database. Hence there is a datasetmoduledata and datasetmodulemeta entries, which lists those properties needed only for modules fetching data, such as the queried object IDs, query state, etc.

Concerning extensibility of the API

The API is highly extensible for both module configuration and database data:

Module configuration: Because modules are nested, the information for one module will never clash with that from another module (for instance, the property “class” for each module will never override the property “class” for another module).

Database data: databases entry contains a relational representation of the data as stored in the database, with objects referencing each other through their ID, so the data for each object is rendered only once:

databases
    primary {
        posts: {
            1: {author: 1, title: "This is a post", ...},
            2: {author: 2, ...},
        },
        users: {
            1: {name: "Leo", ...},
            2: {name: "Pedro", ...},
            ...
        },
        comments: {
            1: {author: 1, post_id: 1, content: "This is a comment by Leo on post 'This is a post'", ...}
        },
        events: {...},
        locations: {...},
    }

This feature can already be observed in PoP’s current implementation, such as under item database in the produced JSON code in this link, which looks like this:

Increasingly complex relationships are possible, yet without ever making the query reach an infinite loop:

Fetch all posts. For each post:
    Fetch the author. For each author:
        Fetch the posts the user has "liked". For each post:
            Fetch all its comments. For each comment:
                Fetch the comment author. For each author:
                    Fetch all the events which this user is attending. For each event: 
                        Fetch its location. For each location: ...

Which translates into object types being loaded like this:

post => user => post => comment => user => event => location => ...

Concerning performance

The API introduces performance considerations for both module configuration and database data:

Module configuration: Splitting the API entries into static and stateful groups makes the configuration highly cacheable: whereas stateful data has state specific to the URL, static data has state specific to the module structure (ie how modules are nested inside each other, from the entry-module to the lowest levels), so it can be re-used for pages with different URLs, such as /events/1/ and /events/2/.

Database data: the retrieved data fields under entry databases are not defined in advance, but are the union of all data fields required by all modules, for all database objects. This model outperforms REST, which has a rigid structure in which all the data fields are defined in advance, and GraphQL, since it similarly fetches all the required data for the page in a single request, but it is cacheable on the server-side.

Concerning reactivity

Because reactivity is server-based, it can’t be implemented through client-based reactive libraries such as React, requiring a completely different approach. Whereas React employs a Virtual DOM to calculate and idenfity the minimal set of operations to apply on an existing element on the DOM to update it, which makes it ideal for highly dynamic operations, Handlebars, on the other side, simply generates the whole view for a module, hence its reactivity is slower. In order to also provide a reasonable reactive behavior, it will need be coupled with vanilla javascript operations and a sensible methodology of when/how to update the DOM:

  • Re-render the layout when:
    • module’s layout configuration was modified
    • module’s database object changed to a different one
  • Re-execute a javascript method when:
    • an input to the javascript method changed

And instead of executing a diff operation to calculate if the state has changed, the response from the API will simply return data whenever it must be re-rendered, under the stateful branch. Hence, the server will know the state of the application (passed back and forth through entry querystate) and return data, or not, accordingly.

In summary, the reactivity produced will not be as highly dynamic as React’s, but it will be good enough to have the site be self-updated after an operation is produced, such as when editing or deleting a post, adding more posts on an infinite-scroll, adding the user’s RSVP to an event, or updating the counter of post likes and user followers.

Concerning data loading

“The module is its own API” concept means that the module does not need be provided an external API to fetch data from (REST, GraphQL), but it can already fetch its own data from the server. The module is self-sufficient, providing rendering and dataloading, and it is isomorphic, providing all the different functionalities from a single codebase. Hence, there is no difference between a module and its API: the module is the API, and the API is the module.

As a consequence, coding in PoP becomes incredibly simple: a single developer can tackle both the layout rendering and dataloading at once, and these two tasks will always work together (for instance, there is no risk of an API not retrieving a field required by the layout).

Once rendered in the client, a module can interact with itself simply by setting its dataloadsource property (i.e. the field indicating from where the module will fetch its data, or execute a POST operation) to the URL of the page where the component was added, plus an entry-module parameter with its module path (i.e. the list of modules, starting from the top level module, until reaching the corresponding module). For instance, requesting URL /some-page/ may produce the following response, bringing database data and module configuration for all modules in the page:

module1
    configuration
    nestedmodules
        module2
            configuration
            nestedmodules
                module3
                    configuration
                module4
                    configuration
                module5
                    configuration
                    nestedmodules
                        module6
                            configuration

And requesting URL /some-page/?entry-module=module1.module2.module5 will produce the following response, bringing database data and configuration for all modules starting only from the module module5, and removing everything else:

module1
    nestedmodules
        module2
            nestedmodules
                module5
                    configuration
                    nestedmodules
                        module6
                            configuration

In addition, a parameter modulefilter can allow to bring a pre-arranged set of modules. For instance, calling a page with ?modulefilter=userstate can print only those modules which require user state for rendering them in the client, such as modules module3 and module6:

module1
    nestedmodules
        module2
            nestedmodules
                module3
                    configuration
                module5
                    nestedmodules
                        module6
                            configuration

Hence, this API delivers better performance than both REST and GraphQL since it can bring more data on each request:

  • REST fetches all data for one entity (eg: one post) or group or similar entities (eg: one set of posts)
  • GraphQL fetches all data for entities of different types in a component (eg: one post plus its author plus the author’s location)
  • PoP fetches all data for all entities, for all components in a page or from a subset of components (eg: all posts, their authors, and their locations, for all components on the page, and with all data retrieved in a relational structure so no data item is duplicated)

Concerning reusability/documentation/maintenance

Using the entry-module parameter to fetch the data for a specific module can also be used to produce the output of that module alone, in isolation, meaning that it will also request the resources (JS/CSS) needed starting from that module onwards. Hence, if requesting /events/ produces the following webpage…

… in which the module for the central calendar is found under the module path toplevel.mainsection.calendar-block.calendar, requesting /events/?entry-module=toplevel.mainsection.calendar-block.calendar will produce the following response:

… and loading the controls module under toplevel.mainsection.calendar-block.calendar.controls we get:

This is the rendering of the module, independent of the context under which it is placed, providing the following benefits:

  • Automatic documentation: the module is self-documented and a living pattern library
  • Transclusion: websites can render other websites’ modules into themselves, and the rendering happens directly in the page, and not in an iframe, as can be already seen in SukiPoP.com (whose feed is rendered with data from several other websites)

Concerning versatility of outputs for different mediums

The nesting of modules allows to make a fork to another module to add compatibility for a specific medium or technology. For instance, let’s say the webpage has this structure…

module1
    configuration
    nestedmodules
        module2
            configuration
            nestedmodules
                module3
                    configuration
                module4
                    configuration

… and we would like to make the website for AMP, however module2 and module4 are not AMP-compatible (eg: they add unsupported Javascript), then we are able to fork the modules into a similar, AMP-compatible modules module2AMP and module4AMP, only for those specific cases, while everything else stays the same:

module1
    configuration
    nestedmodules
        module2AMP
            configuration
            nestedmodules
                module3
                    configuration
                module4AMP
                    configuration

Summary

The new PoP will bring powerful new features. By being based around an open specification, it will open the doors for implementation for other platforms (in addition to PHP/WordPress). The server-based reactivity will provide an alternative to client-based javascript libraries. It will be thoroughly based on components called modules, which provide both their rendering and dataloading from a single codebase. And it will be highly cacheable, both at the server and client-side.

If everything goes fine, the new PoP will be released around end of 2018 or beginning of 2019.


Sign up to our newsletter:

Welcome to the PoP framework!
Break the information monopoly

the PoP framework is open source software which aims to decentralize the content flow and break the information monopoly from large internet corporations. Read more.