Describing the foundations of the new PoP

We are re-architecting PoP to make it much simpler to code with it, faster, and more versatile, planning to release it around end of 2018 or beginning of 2019. The new PoP will feature an architecture based on the following foundations:

  • Everything is a module
  • The module is its own API
  • Reactivity

At the core of these changes is an upgraded API model, which will feature server-based components. Following the example set by GraphQL, which owes its success in part to being a specification instead of an implementation, PoP will be composed of the following layers:

  1. The API (JSON response) specification
  2. PoP Server, to serve content based on the API specification
  3. PoP.js, to consume the content in the client

Being built around an open specification, we can expect different implementations to emerge, for different technologies. The current PoP site will immediately provide the first implementation of PoP Server and PoP.js:

  • PoP Server for WordPress, based on PoP Server for PHP
  • PoP.js through vanilla JS and Handlebars templates

The API (JSON response) specification

In PoP, a component is called a module. Each module is either an atomic functionality, or a composition of other modules. Hence, starting from the lowest-level modules (the atoms), modules will be iteratively composed together until obtaining the highest-level module (the page). The new API model respects the structure of how modules are composed, and print all information needed at each module level.

In PoP, the logic of the application does not reside on the client-side, but on the server-side. The server indicates what javascript functions will be executed, on what elements on the DOM, and with what values. This will extend to reactivity: it is the server who will decide when to make a component be re-rendered. Reactivity is supported by splitting the information for each module into two groups: static, which is non-reactive, and stateful, which is reactive; the application in the server knows if to apply reactivity based on the querystate parameters added to the URL.

The API structure is divided into module configuration, database data, and additional meta entries. It looks like this:

modulesettings:
    static:
        starting from entry-module, iteratively nest {
            configuration
                module
                template
            jsmethods
            dbkeys
            datafields
        }
    stateful:
        starting from entry-module, iteratively nest {
            configuration
        }
moduledata:
    static:
        starting from entry-module, iteratively nest {
            feedback
        }
    stateful:
        starting from entry-module, iteratively nest {
            feedback
        }
datasetmoduledata:
    static:
        starting from entry-module, iteratively nest {
            dbobjectids
        }
    stateful:
        starting from entry-module, iteratively nest {
            dbobjectids
        }
datasetmodulemeta:
    static:
        starting from entry-module, iteratively nest {
            dataloadsource
            lazyload
        }
    stateful:
        starting from entry-module, iteratively nest {
            dataloadsource
            lazyload
            querystate
            queryparams
            queryresult
        }
modulejssettings:
    static:
        starting from entry-module, iteratively nest {
            configuration {
                foreach jsmethod: {...}
            }
        }
    stateful: 
        starting from entry-module, iteratively nest {
            configuration {
                foreach jsmethod: {...}
            }
        }
modulejsdata:
    static:
        starting from entry-module, iteratively nest {
            inputs {
                foreach jsmethod: {...}
            }
        }
    stateful:
        starting from entry-module, iteratively nest {
            inputs {
                foreach jsmethod: {...}
            }
        }
databases
    regular
    userstate
requestmeta
    entry-module
sessionmeta
    loggedinuser
sitemeta
    sitetitle

Let’s analyze the features provided by the API’s structure:

Concerning modules: the “Everything is a module” maxim implies at there will only be 1 module at the top of the page, from which we start composing all the other modules. This module is defined under entry-module. Then, rendering the page will simply be rendering the entry-module:

Concerning data: Not all modules fetch data from the database. Hence there is a datasetmoduledata and datasetmodulemeta entries, which lists those properties needed only for modules fetching data, such as the queried object IDs, query state, etc.

Concerning extensibility of the API: The API is highly extensible for both module configuration and database data:

Module configuration: Because modules are nested, the information for one module will never clash with that from another module (for instance, the property “class” for each module will never override the property “class” for another module).

Database data: databases entry contains a relational representation of the data as stored in the database, with objects referencing each other through their ID, so the data for each object is rendered only once:

Increasingly complex relationships are possible, yet without ever making the query reach an infinite loop:

Fetch all posts. For each post:
    Fetch the author. For each author:
        Fetch the posts the user has "liked". For each post:
            Fetch all its comments. For each comment:
                Fetch the comment author. For each author:
                    Fetch all the events which this user is attending. For each event: 
                        Fetch its location. For each location: ...

Which translates into object types being loaded like this:

post => user => post => comment => user => event => location => ...

Concerning performance: The API introduces performance considerations for both module configuration and database data:

Module configuration: Splitting the API entries into static and stateful groups makes the configuration highly cacheable: whereas stateful data has state specific to the URL, static data has state specific to the module structure (ie how modules are nested inside each other, from the entry-module to the lowest levels), so it can be re-used for pages with different URLs, such as /events/1/ and /events/2/.

Database data: the retrieved data fields under entry databases are not defined in advance, but are the union of all data fields required by all modules, for all database objects. This model outperforms REST, which has a rigid structure in which all the data fields are defined in advance, and GraphQL, since it similarly fetches all the required data for the page in a single request, but it is cacheable on the server-side.

Concerning reactivity: Because reactivity is server-based, it can’t be implemented through client-based reactive libraries such as React, requiring a completely different approach. Whereas React employs a Virtual DOM to calculate and idenfity the minimal set of operations to apply on an existing element on the DOM to update it, which makes it ideal for highly dynamic operations, Handlebars, on the other side, simply generates the whole view for a module, hence its reactivity is slower. In order to also provide a reasonable reactive behavior, it will need be coupled with vanilla javascript operations and a sensible methodology of when/how to update the DOM:

  • Re-render the layout when:
    • module’s layout configuration was modified
    • module’s database object changed to a different one
  • Re-execute a javascript method when:
    • an input to the javascript method changed

And instead of executing a diff operation to calculate if the state has changed, the response from the API will simply return data whenever it must be re-rendered, under the stateful branch. Hence, the server will know the state of the application (passed back and forth through entry querystate) and return data, or not, accordingly.

In summary, the reactivity produced will not be as highly dynamic as React’s, but it will be good enough to have the site be self-updated after an operation is produced, such as when editing or deleting a post, adding more posts on an infinite-scroll, adding the user’s RSVP to an event, or updating the counter of post likes and user followers.

Concerning isomorphism of code: “The module is its own API” concept means that the module does not need be provided an external API to fetch data from (REST, GraphQL), but it can already fetch its own data from the server. The module is self-sufficient, providing rendering and dataloading, and it is isomorphic, providing all the different functionalities from a single codebase. Hence, there is no difference between a module and its API: the module is the API, and the API is the module.

As a consequence, coding in PoP becomes incredibly simple: a single developer can tackle both the layout rendering and dataloading at once, and these two tasks will always work together (for instance, there is no risk of an API not retrieving a field required by the layout).

Once rendered in the client, a module can interact with itself simply by setting its dataloadsource property (i.e. the field indicating from where the module will fetch its data, or execute a POST operation) to the URL of the page where the component was added, plus an entry-module parameter with its module path (i.e. the list of modules, starting from the top level module, until reaching the corresponding module). For instance, requesting URL /some-page/ may produce the following response, bringing database data and module configuration for all modules in the page:

module1
    configuration
    nestedmodules
        module2
            configuration
            nestedmodules
                module3
                    configuration
                module4
                    configuration
                module5
                    configuration
                    nestedmodules
                        module6
                            configuration

And requesting URL /some-page/?entry-module=module1.module2.module5 will produce the following response, bringing database data and configuration for all modules starting only from the module module5, and removing everything else:

module1
    nestedmodules
        module2
            nestedmodules
                module5
                    configuration
                    nestedmodules
                        module6
                            configuration

Summary

The new PoP will bring powerful new features. By being based around an open specification, it will open the doors for implementation for other platforms (in addition to PHP/WordPress). The server-based reactivity will provide an alternative to client-based javascript libraries. It will be thoroughly based on components called modules, which provide both their rendering and dataloading from a single codebase. And it will be highly cacheable, both at the server and client-side.

If everything goes fine, the new PoP will be released around end of 2018 or beginning of 2019.


Sign up to our newsletter:

Welcome to the PoP framework!
Break the information monopoly

the PoP framework is open source software which aims to decentralize the content flow and break the information monopoly from large internet corporations. Read more.