Manuel Shimano Recommendations

on 7 Jun, 18:19

Notes on PoP’s code migration to Composer

Notes on PoP’s code migration to Composer
Blog

We have finally started migrated PoP to Composer components, hence it will soon be possible to install it with a simple `composer require` command. We are expecting this task to be finished around end of 2019. The migration involves the following actions:

All components already migrated can be found under PoP’s account in GitHub: github.com/getpop. To date, we have migrated the following components:

  • root: Declaration of dependencies shared by all PoP components
  • engine-wp: Implementation of PoP Engine for WordPress
  • engine-wp-bootloader: PoP Engine Bootloader for WordPress
  • engine-symfony: Implementation of contracts for PoP Engine through Symfony components
  • hooks: Contracts to implement hooks (filters and actions) for PoP
  • hooks-wp: WordPress implementation of the contracts to implement hooks (filters and actions) for PoP
  • translation: Translation API for PoP components
  • translation-wp: Implementation of the Translation API for WordPress
  • cache: Contracts for implementing caching for PoP components
  • cache-symfony: Implementation of the contracts for caching through Symfony Cache
  • filestore: Base classes to save files to disk, read them and reference them through an URL
  • spa: Single-Page Application for PoP

In addition, we have created a Bootstrap project for WordPress, enabling to quickly install WordPress together with PoP. This project will contain the most basic PoP components, which will be continuously added as their migration is completed.

The most important component in PoP is engine, since it contains the fundamental logic for rendering components in PoP. This component is currently being migrated. Once its migration is complete, PoP’s fundamentals will be in place, so that any application will be able to use it. From then on, the other components to migrate will be features.

The migration is taking a very long time due to the following reasons:

  • PoP’s code was based on WordPress, which is very old. Hence, we’re conveniently converting PoP to a modern codebase, based around components installed through Composer, and full compatibility with the PHP Standard Recommendations, particularly PSRs 1, 2 and 4
  • During the previous 5 years, we developed tons of functionalities which must be migrated to the modern codebase

How many components must still be migrated for the migration to be completed? A lot:

If anyone is unwilling to wait for so long until the migration is complete, and want to contribute with the migration, give Leo a shout!

on 28 Oct, 16:45

Describing the foundations of the new PoP

Describing the foundations of the new PoP
Blog

We are re-architecting PoP to make it much simpler to code with it, faster, and more versatile, planning to release it around end of 2018 or beginning of 2019. The new PoP will feature an architecture based on the following foundations:

  • Everything is a module
  • The module is its own API
  • Reactivity

At the core of these changes is an upgraded API model, which will feature server-based components. Following the example set by GraphQL, which owes its success in part to being a specification instead of an implementation, PoP will be composed of the following layers:

  1. The API (JSON response) specification
  2. PoP Server, to serve content based on the API specification
  3. PoP.js, to consume the content in the client

Being built around an open specification, we can expect different implementations to emerge, for different technologies. The current PoP site will immediately provide the first implementation of PoP Server and PoP.js:

  • PoP Server for WordPress, based on PoP Server for PHP
  • PoP.js through vanilla JS and Handlebars templates

The API (JSON response) specification

In PoP, a component is called a module. Each module is either an atomic functionality, or a composition of other modules. Hence, starting from the lowest-level modules (the atoms), modules will be iteratively composed together until obtaining the highest-level module (the page). The new API model respects the structure of how modules are composed, and print all information needed at each module level.

In PoP, the logic of the application does not reside on the client-side, but on the server-side. The server indicates what javascript functions will be executed, on what elements on the DOM, and with what values. This will extend to reactivity: it is the server who will decide when to make a component be re-rendered. Reactivity is supported by splitting the information for each module into two groups: static, which is non-reactive, and stateful, which is reactive; the application in the server knows if to apply reactivity based on the querystate parameters added to the URL.

The API structure is divided into module configuration, database data, and additional meta entries. Once finished, it will look something like this:

modulesettings:
    static:
        foreach module {
            configuration
                module
                template
            jsmethods
            dbkeys
            datafields
            foreach nestedmodules {
                repeat ...
            }
        }
    stateful:
        foreach module {
            configuration
            foreach nestedmodules {
                repeat ...
            }
        }
moduledata:
    static:
        foreach module {
            feedback
            foreach nestedmodules {
                repeat ...
            }
        }
    stateful:
        foreach module {
            feedback
            foreach nestedmodules {
                repeat ...
            }
        }
datasetmoduledata:
    static:
        foreach module {
            dbobjectids
            foreach nestedmodules {
                repeat ...
            }
        }
    stateful:
        foreach module {
            dbobjectids
            foreach nestedmodules {
                repeat ...
            }
        }
datasetmodulemeta:
    static:
        foreach module {
            dataloadsource
            lazyload
            foreach nestedmodules {
                repeat ...
            }
        }
    stateful:
        foreach module {
            dataloadsource
            lazyload
            querystate
            queryparams
            queryresult
            foreach nestedmodules {
                repeat ...
            }
        }
modulejssettings:
    static:
        foreach module {
            configuration {
                foreach jsmethod: {...}
            }
            foreach nestedmodules {
                repeat ...
            }
        }
    stateful: 
        foreach module {
            configuration {
                foreach jsmethod: {...}
            }
            foreach nestedmodules {
                repeat ...
            }
        }
modulejsdata:
    static:
        foreach module {
            inputs {
                foreach jsmethod: {...}
            }
            foreach nestedmodules {
                repeat ...
            }
        }
    stateful:
        foreach module {
            inputs {
                foreach jsmethod: {...}
            }
            foreach nestedmodules {
                repeat ...
            }
        }
databases
    primary {
        ...
    }
    userstate {
        ...
    }
requestmeta
    entry-module
sessionmeta
    loggedinuserdata
sitemeta
    sitetitle

Let’s analyze the features provided by the API’s structure:

Concerning modules

The “Everything is a module” maxim implies at there will only be 1 module at the top of the page, from which we start composing all the other modules. This module is defined under entry-module. Then, rendering the page will simply be rendering the entry-module:

Concerning data

Not all modules fetch data from the database. Hence there is a datasetmoduledata and datasetmodulemeta entries, which lists those properties needed only for modules fetching data, such as the queried object IDs, query state, etc.

Concerning extensibility of the API

The API is highly extensible for both module configuration and database data:

Module configuration: Because modules are nested, the information for one module will never clash with that from another module (for instance, the property “class” for each module will never override the property “class” for another module).

Database data: databases entry contains a relational representation of the data as stored in the database, with objects referencing each other through their ID, so the data for each object is rendered only once:

databases
    primary {
        posts: {
            1: {author: 1, title: "This is a post", ...},
            2: {author: 2, ...},
        },
        users: {
            1: {name: "Leo", ...},
            2: {name: "Pedro", ...},
            ...
        },
        comments: {
            1: {author: 1, post_id: 1, content: "This is a comment by Leo on post 'This is a post'", ...}
        },
        events: {...},
        locations: {...},
    }

This feature can already be observed in PoP’s current implementation, such as under item database in the produced JSON code in this link, which looks like this:

Increasingly complex relationships are possible, yet without ever making the query reach an infinite loop:

Fetch all posts. For each post:
    Fetch the author. For each author:
        Fetch the posts the user has "liked". For each post:
            Fetch all its comments. For each comment:
                Fetch the comment author. For each author:
                    Fetch all the events which this user is attending. For each event: 
                        Fetch its location. For each location: ...

Which translates into object types being loaded like this:

post => user => post => comment => user => event => location => ...

Concerning performance

The API introduces performance considerations for both module configuration and database data:

Module configuration: Splitting the API entries into static and stateful groups makes the configuration highly cacheable: whereas stateful data has state specific to the URL, static data has state specific to the module structure (ie how modules are nested inside each other, from the entry-module to the lowest levels), so it can be re-used for pages with different URLs, such as /events/1/ and /events/2/.

Database data: the retrieved data fields under entry databases are not defined in advance, but are the union of all data fields required by all modules, for all database objects. This model outperforms REST, which has a rigid structure in which all the data fields are defined in advance, and GraphQL, since it similarly fetches all the required data for the page in a single request, but it is cacheable on the server-side.

Concerning reactivity

Because reactivity is server-based, it can’t be implemented through client-based reactive libraries such as React, requiring a completely different approach. Whereas React employs a Virtual DOM to calculate and idenfity the minimal set of operations to apply on an existing element on the DOM to update it, which makes it ideal for highly dynamic operations, Handlebars, on the other side, simply generates the whole view for a module, hence its reactivity is slower. In order to also provide a reasonable reactive behavior, it will need be coupled with vanilla javascript operations and a sensible methodology of when/how to update the DOM:

  • Re-render the layout when:
    • module’s layout configuration was modified
    • module’s database object changed to a different one
  • Re-execute a javascript method when:
    • an input to the javascript method changed

And instead of executing a diff operation to calculate if the state has changed, the response from the API will simply return data whenever it must be re-rendered, under the stateful branch. Hence, the server will know the state of the application (passed back and forth through entry querystate) and return data, or not, accordingly.

In summary, the reactivity produced will not be as highly dynamic as React’s, but it will be good enough to have the site be self-updated after an operation is produced, such as when editing or deleting a post, adding more posts on an infinite-scroll, adding the user’s RSVP to an event, or updating the counter of post likes and user followers.

Concerning data loading

“The module is its own API” concept means that the module does not need be provided an external API to fetch data from (REST, GraphQL), but it can already fetch its own data from the server. The module is self-sufficient, providing rendering and dataloading, and it is isomorphic, providing all the different functionalities from a single codebase. Hence, there is no difference between a module and its API: the module is the API, and the API is the module.

As a consequence, coding in PoP becomes incredibly simple: a single developer can tackle both the layout rendering and dataloading at once, and these two tasks will always work together (for instance, there is no risk of an API not retrieving a field required by the layout).

Once rendered in the client, a module can interact with itself simply by setting its dataloadsource property (i.e. the field indicating from where the module will fetch its data, or execute a POST operation) to the URL of the page where the component was added, plus an entry-module parameter with its module path (i.e. the list of modules, starting from the top level module, until reaching the corresponding module). For instance, requesting URL /some-page/ may produce the following response, bringing database data and module configuration for all modules in the page:

module1
    configuration
    nestedmodules
        module2
            configuration
            nestedmodules
                module3
                    configuration
                module4
                    configuration
                module5
                    configuration
                    nestedmodules
                        module6
                            configuration

And requesting URL /some-page/?entry-module=module1.module2.module5 will produce the following response, bringing database data and configuration for all modules starting only from the module module5, and removing everything else:

module1
    nestedmodules
        module2
            nestedmodules
                module5
                    configuration
                    nestedmodules
                        module6
                            configuration

In addition, a parameter modulefilter can allow to bring a pre-arranged set of modules. For instance, calling a page with ?modulefilter=userstate can print only those modules which require user state for rendering them in the client, such as modules module3 and module6:

module1
    nestedmodules
        module2
            nestedmodules
                module3
                    configuration
                module5
                    nestedmodules
                        module6
                            configuration

Hence, this API delivers better performance than both REST and GraphQL since it can bring more data on each request:

  • REST fetches all data for one entity (eg: one post) or group or similar entities (eg: one set of posts)
  • GraphQL fetches all data for entities of different types in a component (eg: one post plus its author plus the author’s location)
  • PoP fetches all data for all entities, for all components in a page or from a subset of components (eg: all posts, their authors, and their locations, for all components on the page, and with all data retrieved in a relational structure so no data item is duplicated)

Concerning reusability/documentation/maintenance

Using the entry-module parameter to fetch the data for a specific module can also be used to produce the output of that module alone, in isolation, meaning that it will also request the resources (JS/CSS) needed starting from that module onwards. Hence, if requesting /events/ produces the following webpage…

… in which the module for the central calendar is found under the module path toplevel.mainsection.calendar-block.calendar, requesting /events/?entry-module=toplevel.mainsection.calendar-block.calendar will produce the following response:

… and loading the controls module under toplevel.mainsection.calendar-block.calendar.controls we get:

This is the rendering of the module, independent of the context under which it is placed, providing the following benefits:

  • Automatic documentation: the module is self-documented and a living pattern library
  • Transclusion: websites can render other websites’ modules into themselves, and the rendering happens directly in the page, and not in an iframe, as can be already seen in SukiPoP.com (whose feed is rendered with data from several other websites)

Concerning versatility of outputs for different mediums

The nesting of modules allows to make a fork to another module to add compatibility for a specific medium or technology. For instance, let’s say the webpage has this structure…

module1
    configuration
    nestedmodules
        module2
            configuration
            nestedmodules
                module3
                    configuration
                module4
                    configuration

… and we would like to make the website for AMP, however module2 and module4 are not AMP-compatible (eg: they add unsupported Javascript), then we are able to fork the modules into a similar, AMP-compatible modules module2AMP and module4AMP, only for those specific cases, while everything else stays the same:

module1
    configuration
    nestedmodules
        module2AMP
            configuration
            nestedmodules
                module3
                    configuration
                module4AMP
                    configuration

Summary

The new PoP will bring powerful new features. By being based around an open specification, it will open the doors for implementation for other platforms (in addition to PHP/WordPress). The server-based reactivity will provide an alternative to client-based javascript libraries. It will be thoroughly based on components called modules, which provide both their rendering and dataloading from a single codebase. And it will be highly cacheable, both at the server and client-side.

If everything goes fine, the new PoP will be released around end of 2018 or beginning of 2019.

on 23 Feb, 19:07

We will soon work on making PoP easy to install

We will soon work on making PoP easy to install
Blog

PoP is great, but there is only one (not so tiny) problem: it is not easy to install. The reason for that is that, until today, Leo has never had the opportunity to work on the scripts to automate installation since there were other priorities.

We are currently creating scripts to automate the installation process, we expect them to be ready around mid-October 2018.

Until then, we provide a zip file including all code (PoP, WordPress and plugins), and a database dump from the GetPoP Demo website, to set-up this same site in a quick-and-dirty manner in your localhost. Download the files and read the installation instructions here.

If you run into any trouble, please do not hesitate to contact Leo.

on 10 Feb, 06:17

Native support for A/B Testing in PoP

Native support for A/B Testing in PoP
Blog

PoP has support for A/B Testing natively, through the modification of one or two URL parameters which change the look and feel of a webpage:

  • thememode
  • themestyle

Whereas thememode is used for rendering a webpage for a completely different use case (such as embed, print, or mobile), themestyle is used to tweak the presentation styles. Some examples:

Homepage:

Homepage for printing purposes, appending thememode=print to the URL:

Homepage with a wider presentation, appending themestyle=expansive to the URL:

Once the website is initially loaded with a given value for any of these two parameters, the framework in the front-end will always append these values to all subsequent requests. In the image below, themestyle=expansive is added to all requested URL, either as a link clicked upon by the user, or dynamic content fetched by some component:

Then, a PoP site can be customized using these parameters, and users can be served either version. Because this information will always be in the requested URL, then we can analyze and measure the impacts of each design very easily. Finally, to append one or another parameter value to the URL, the URL is intercepted in the client through Service Workers, adding a parameter or the other following the weight given to each.

This is native A/B Testing.

on 7 Feb, 16:30

Improved performance: FMP < 1 sec, SpeedIndex < 1250

Improved performance: FMP < 1 sec, SpeedIndex < 1250
Blog

After deploying PoP’s latest feature, dynamic bundling of resources, PoP sites experience a wonderful performance boost in their initial load. For instance, site Agenda Urbana is now scoring a sweet 1164 Speed Index score and rendering the first meaningful paint (FMP) in just 0.8 seconds, both below the recommended goal of 1250 for Speed Index and under 1 second for FMP:

Having the lowest possible loading time for our sites has become priority #1, more since Google announced that, starting from July 2018, page speed will be a ranking factor for mobile searches.

Because of this, we will keep working hard on producing the most optimized website ever. We plan to lower the Speed Index score even further with the following developments:

  • Add all assets loaded by WordPress itself (eg: jQuery, others) into the bundles (currently they are not)
  • Split Bootstrap into all of its components, to be loaded as needed through code-splitting
  • Route content through a CDN also when initially loading the site
  • Use IntersectionObserver for lazy-loading images and producing infinite-scrolling (the latter one will remove the dependency on library Waypoints)
  • Enable images to be loaded from an image optimizer service (such as Cloudinary)

As usual, enjoy!

on 12 Dec, 02:11

Coming soon: GutenPoP, the integration of WP Gutenberg with PoP

Coming soon: GutenPoP, the integration of WP Gutenberg with PoP
Blog

WordPress Gutenberg is looking incredibly impressive, as it can be appreciated in the demo provided during WordCamp US 2017. Gutenberg will change drastically how WordPress sites are built with its intuitive, yet extremely powerful, user interface. While currently Gutenberg allows to only create content (replacing the content editor), in the future it is expected to be able to construct the whole site. In this post, Morten Rand-Hendriksen explains the new approach quite clearly: “Today we think of creating posts and pages. In the WordPress of Tomorrow, we will be making blocks and placing them in views.” This is how the WordPress of tomorrow will look like:

gutenberg-blocks

Blocks. Inside the views. Remarkably, that is what PoP is all about! Now, look how blocks are already filling the view in the PoP of today:

blocks

Also remarkably, these components have also been called blocks in PoP, putting in evidence that the Gutenberg concept is the same PoP uses. Whereas the WordPress community has been working on adding the block-concept to WordPress since 11 months ago, PoP has been based on this concept since its inception, already 4.5 years ago!

PoP, however, takes the block concept deeper down: the smallest units of components in a PoP view are not blocks, but modules, which are either an atomic representation of a functionality (eg: something as basic as a link), or a composition of other modules. It looks like this:

modules

In PoP, everything is a module. Blocks are also modules, with the particularity that they have their own state, and they know from what URL to fetch their data. A block can be as simple as containing 1 inner module, or extremely complex. Moreover, blocks can be nested, wrapping other blocks, creating incredibly powerful components, yet made-up of re-using independent, autonomous ones. It looks like this:

blockgroups

In this image above, the block with text “Welcome to PoP Demo!” is wrapping another block, which adds the gray background styling, and which itself is wrapping 2 other blocks: a log-in menu, and a newsletter form. These 2 blocks are completely autonomous, and they live in the website in different other views too. When nesting, blocks can alter the attributes of the inner blocks, thanks to PoP’s top-down approach for setting properties.

What’s coming next

Given all the similarities between Gutenberg and PoP, including how they are both based on block populating the view, it is extremely compelling to provide an integration between PoP and Gutenberg. PoP could perfectly augment the capabilities of a Gutenberg-based website by providing its own blocks as Gutenberg blocks.

In the following months, we will work on these features:

  • Site-builder integration: use Gutenberg to insert PoP blocks into the view of a GutenPoP website
  • Front-end integration: add Gutenberg to the front-end of a PoP website, for users to be able to create content (posts, comments) using the Gutenberg interface

We are extremely excited with this development, we can see how the combination of Gutenberg and PoP will enable the creation of highly-dynamic websites, yet in a purely drag-and-drop, users-can-do-it-from-the-frontend manner.


Sign up to our newsletter:

Welcome to the PoP framework!
Break the information monopoly

the PoP framework is open source software which aims to decentralize the content flow and break the information monopoly from large internet corporations. Read more.