Superfeedr Blog

Updated Pricing

2017-02-06T00:00:00-05:00

Today, we are introducing a new pricing for Superfeedr’s subscribers.

In our early days, we wanted Superfeedr’s pricing to be based on the amount of data our service would send you. As we were very young start-up, we made the assumption that verbose feeds would cost us more to process and that it would be fair to pass this cost down to customers. In retrospect, this was not the best approach as it means that our pricing was less predictable than it should be: subscribers do not control how verbose the feeds they’re subscribed to are.

As the Google Feed API is gone, a lot of our most recent customer are only using our “pull” APIs. This means that they often have no visibility in the number of update for any given feed. This is another reason to reconsider our pricing.

Our new approach is to charge our subscribers based on the number of feeds we process on their behalf. We analysed subscriber accounts and the recent invoices and came up with this beautiful scatter graph which allowed us to determine what would be the right breakdown.

And here are the rules we will be applying to all new subscriber accounts:

10 cts per subscription for the first 50 feeds,
5 cts for each feed up to 5,000 feeds,
2 cts for each feed up to 50,000 feeds,
1 ct per feed for each feed beyond that.

You can also use our simulator for a more detailed estimation.

Additionally, if your invoice is less than $1, we will waive it; this means that the first 10 feeds are always free.

Finally, even though this pricing should almost always mean lower invoices for our existing customers, we offer them the ability to remain on the current pricing for as long as they want.

Google Feed API is gone: now what?

2016-12-12T00:00:00-05:00

As you might have heard, the Google Feed API is closing on December 15th, 2016. It’s not too late to look for a replacement and Superfeedr has you covered, either directly, or via Embedly.

The Google Feed API was used to build all types of applications, from simple web page embeds to more advanced feed readers and podcasts players. Superfeedr is a powerful replacement which also provides tools for all of these use cases. We also have solutions for feed publishers or even for people looking for real-time search filters.

Compared to the Google Feed API, we provide a richer schema (with more metadata!).

Push vs. Pull

One of the largest benefits of using Superfeedr’s infrastructure is that rather than polling data from us, you can subscribe to the feeds and be notified using webhooks when these are updated. That means you do not have to load data from Superfeedr on your web pages, but can conveniently store it on your server and serve it from there.

Embedding feeds

In addition to consuming feeds, if you’re looking to embed feeds, consider using Embedly. Embedly is a Superfeedr customer and the processing of feeds is handled transparently by us.

Their embed is very simple to configure and is responsive which means it will look great on both desktop and mobile web. Here’s an example with the feed for this blog (you can replace the value of the href attribute with any feed):

<a class="embedly-card" href="https://superfeedr-blog-feed.herokuapp.com/">Feed</a>
<script async src="//cdn.embedly.com/widgets/platform.js" charset="UTF-8"></script>

And here is the output:

Feed

Building a News Bot for Facebook Messenger

2016-11-01T00:00:00-04:00

With each month that goes by, there is less and less doubt that messaging applications and bots are one of the interfaces with whom we are interacting more and more. One of the verticals for which messaging applications are the most useful is news.

We built a news bot Facebook Messenger which lets you follow you favorite websites and be notified when they publish new stories.

Try it first

If you have a Facebook account, you should start by testing the news bot to see what it’s capable of. It’s as easy as opening this link. You should now be able to start a conversation with Superfeedr’s bot. First I usually say Hello.

The bots let you then type (or paste really), the URL of your favorite site. I chose Superfeedr’s Medium publication so I typed http://medium.com/superfeedr-thoughts. The bot then uses Feediscovery to find an relevant RSS feed and asks for a final confirmation. Once this is all set, the next time the publication publishes something, you should get a message from the bot to read that story.

This is a pretty simple bot but already does wonders!

How we built it

The quick answer is that we took the exact same code base that we used for Telegram bot we built last year. It also runs on Amazon’s Lambda infrastructure which, even if it is complex to set up is perfect of webhook based services.

The whole bot code is available on Github.

Setting things up on Facebook.

First, we have to build a Facebook application. You also need a Facebook page as Facebook’s Messenger bots are linked to a page.

Switch to the Messenger settings and start by generating a token that you’ll use to send requests to the Facebook API. For this, you need to select the page that users will interact with. Copy this token to the settings in the init file.

Then, you have to configure the webhooks for incoming messages.

The base URL is where our service will be deployed (you should use an Amazon Lambda URL if you’re deploying your bot there). Our code routes incoming webhooks using the platform query string parameter.

We also specify the types of messages we want to receive: message_deliveries, messages, messaging_optins, messaging_postbacks.

Facebook will send a verification of intent request (PubSubHubbub FTW!) and you should set a random verification token: the bot’s code should handle that by default.

After that you should be all set.

Feel free to follow the quick start instructions in Facebook’s docs for more details.

Architecture overview

The news bot code is routing incoming events based on the platform query string value. Each message is sent to the right platform handler to be parsed. Each of these platform handlers (Facebook and Telegram for now) implements a parseChatMessage method which yields back to the main router.

The main router then identifies the command. We currently support the following commands: hello, help, list, subscribe, unsubscribe, url and version. Once they have been executed; they themselves call the platforms to send back messages to users. By using this, we can let each platform implement its own specific features, such as buttons or custom keyboards.

When a user starts to follow a feed, we issue a subscription request to Superfeedr. The callback URL used for these subscriptions includes the platform as well as a unique chatId to identify the exact user who should receive the data. By doing this, we do not have to store any state inside our bot, making the webhook pattern trivial to scale.

Finally, when Superfeedr sends a notification to the news bot, we extract the platform and chat id from the callback URL, as well as the content of the message and invoke the platform’s sendMessage method.

The future

Adding support for more platforms such as Slack or Kik should be fairly simple. We could also start adding more commands such as track to implement the tracker capabilities of Superfeedr.

Please, let us know what you think! You’re more than welcome to fork the code!

Medium acquires Superfeedr

2016-06-02T00:00:00-04:00

The web is very different from what it was 8 years ago. We’ve said it several times: publishing and consuming content are new frontiers for most of the web giants like Facebook, Google or Apple. We consume the web from mobile devices, we discover content on silo-ed social networks and, more importantly, the base metaphor for the web is shifting from “space” to “time”.

Superfeedr, the open web’s leading feed API and PubSubHubbub hub has been an independent player for 8 years. Superfeedr exists in order to enable people to exchange information on the web more freely and easily. Today, we’re excited to announce Superfeedr has been acquired by Medium. In many ways, it’s a very natural fit: Medium wants to create the best place to publish, distribute and consume content on the web. Together, we are hoping to keep Medium the company a leader in good industry practices, and Medium the network a place where this conversation can gain even more traction.

At Superfeedr, we’ve promoted the open web by embracing open formats and protocols, such as RSS, Atom, XMPP and PubSubHubbub. Over the years, we’ve also learned that these protocols are only as powerful as the people who promote them. As we want the open web to remain strong, we were delighted to find that the folks at Medium share the same values. And we think that Superfeedr’s acquisition is a powerful indicator of Medium’s support for open protocols. As an example of this, my first commit at Medium was to enable full content RSS feeds both for publications and users (available in your settings). Medium already lets you add your own domain name, import and export all of your posts, point to a canonical URL if you cross-posted, supports DNT… and we’ve started working on more!

If you are a Superfeedr customer, nothing has changed: it’s business as usual. More than ever, we are dedicated to providing you with a convenient way to consume or publish feeds from all over the web using open protocols as well as tracking keywords in real time.

Finally, I also want to take a couple moments to thank all the Superfeedr family: the numerous contractors with whom we’ve worked over the years, our investors (Betaworks and Mark Cuban), as well as our advisors and partners.

I’m confident Medium is the right home for Superfeedr. The team here is one of the most talented I’ve ever met and we’re already leading the way when it comes to enabling amazing content publishing. Every month, people collectively spend 2,000,000 hours reading these stories, and I’m delighted to be part of the team building this!

Focus on the Front-End: Building an Asset Repository

2016-04-05T00:00:00-04:00

Change is afoot in Superfeedr’s websites—change in the form of patterns, centralization, and standardization.

In the summer of 2014, a team from Van Patten Media Inc.—in this case, Chris Van Patten and myself—worked with Julien here at Superfeedr to redesign the application’s web interface. We updated a dated design, surfacing functionality that was previously hidden, and made it easier for people new to Superfeedr to figure out just how great it is. The Van Patten Media team also worked on the public-facing elements of the website, redesigning the marketing site, documentation, and blog from the ground up.

We were happy with the results of these redesign projects, and were excited at the opportunity to work with Julien again on a new project: developing a centralized asset repository and pattern library for all Superfeedr web properties.

Project goals

We had a few key goals with this latest project:

Develop a repository that all properties could pull from, centrally-updated so that improvements were shared;
Cut down on redundancy and push for more consistency across the interfaces;
Document all the patterns in use across Superfeedr’s various sites;
Make it easier to design new pages or properties in the future.

We’re happy to say that, after a few weeks of work, we’ve accomplished these goals! With all these new changes, there are a few in particular that we want to highlight and dig into a little further.

Patterns, patterns, patterns

Whether you call it a pattern library or a style guide, this method of development and documentation has increased in popularity over the last few years, and for good reason: it encourages and rewards modular thinking in terms of markup, styles, and functionality for front-end components.

When we set out on our initial redesign of Superfeedr back in in 2014, our first code took place in a small patterns.html file. We didn’t set out to create a proper “pattern library” on day one, but having this file made it easier to see a collection of components in one place and, as we started to design full pages, simplified the work by letting us copy and paste.

This initial, informal pattern library was brought out again as we started working on this most recent project. Because a key goal for the project was to create a “centralised repo” for our assets, we knew we’d have to combine stylesheets from a few sources. A structured, formal pattern library would help prevent us from missing key modules or duplicating code as we merged the assets into one canonical source.

We evaluated a number of open source pattern documentation tools, but eventually landed on a custom approach. We gave each pattern its own HTML file, and loaded each one into an <iframe> in the final output via a build process powered by gulp-file-include, using the srcdoc attribute. This ensured that each pattern could function in isolation, unaffected by other modules or the pattern library itself.

We also added a brief description and usage notes for each pattern, so that anyone looking to extend Superfeedr in the future could easily understand which patterns to use, and how. We built, in essence, a tiny Bootstrap for Superfeedr’s use—and you can view it yourself, too!

Making this pattern library helped us find parts of the design that were redundant, buggy, or less than ideal, and fix them to a known-good standard—fitting, given Superfeedr’s huge support for standards-based applications! Even though Superfeedr is just one entity, its four properties now stand that much more unified thanks to the documentation and implementation of these patterns, and any future expansions of the service can be quickly designed and implemented.

Bootstrap compatibility

When Julien developed river.news, he wanted to show off Superfeedr’s capabilities in an application that could be plugged into many websites and just work. When he did this, he styled the markup with Bootstrap classes, knowing that many websites use Bootstrap, so that if River.news was embedded on a site already using Bootstrap, it’d look great. However, he also wanted code styled with Bootstrap classes displayed on a Superfeedr property to look and feel like Superfeedr.

After evaluating a few options, and some conceptual tug-of-war as we tried to determine the most efficient way to do this, we settled on Sass @extends, which allow you to apply the styles from one CSS rule to another.

For example, here’s how we mapped Bootstrap’s inimitable button classes:

.btn {
  @extend .button;
}

  .btn-default,
  .btn-info,
  .btn-primary,
  .btn-success,
  .btn-warning,
  .btn-danger {
    @extend .button--raised;
  }

  .btn-default,
  .btn-info {
    @extend .button--neutral;
  }

  .btn-primary,
  .btn-success {
    @extend .button--positive;
  }

…and so on.

As you can see, we were able to avoid duplicating the actual rules by mapping our existing styles to the Bootstrap class names.

As we worked on this feature, we had a number of discussions (between ourselves and with Julien) about this particular requirement. It’s an interesting challenge: how do you make an embeddable module look “native” in as many contexts as possible? We settled on this Bootstrap “map”, but there are other approaches as well. We like the look of CSS modules, which allow you to automatically namespace and modularise CSS within a given widget. CSS modules are designed for large CSS codebases, but could also allow projects like River.news to provide “mappings” to frameworks like Bootstrap or Foundation, and allow users to swap between them without requiring multiple sets of markup. It would also make it easier for users to provide custom CSS that fits within their unique design.

We also realised the importance of good documentation of classes and markup. Even if not using a standardized CSS class structure—like BEM, OOCSS, SMACSS, or so on—adopting standard conventions for HTML structure and class names in your project can make it easier for others to adapt your work to their particular use case, without needing to fork it or reinvent the wheel.

Markup underlies all the work we do on the web, and being sensible about its form will go a long way towards increasing the interoperability of our various components.

Conclusion

It has been a thrill to work with Julien on a service that’s so important to so many organisations and projects around the web. Superfeedr stands for open protocols, open software, and open communications—which is why we were so excited to write this post and document our efforts.

Working with Julien on Superfeedr has been a fantastic experience, and we’re proud of the results of this project: we created a pattern library and central assets repository that drives all of the Superfeedr web properties, and will make it much easier to update them or make new ones in the future. We’re excited to see where Julien takes Superfeedr next!

Embed and customize River.news

2016-03-30T00:00:00-04:00

The superfeedr feed API is quite powerful and lets anyone build any kind of application which consumes RSS feeds, whether it’s on the web, a mobile application, or processed on the server side. Building and maintaining servers is not always trivial, which is why more and more applications are built to be server-less. This is the case of our river.news app. We recently updated it and here’s a breakdown of what it does.

Not only you can use River.news on its own domain, but you can also effortlessly embed on any HTML page… like this one:

Offline: Service Workers

If you read this blog often, you’ve already seen that we added offline support to river.news. Service Workers are completely changing the web from a document-oriented architecture (HTML pages are loaded first) to an application-oriented architecture with javascript executed even before HTTP requests are fired.

Better styling

Recently, we centralised all of our styling assets on assets.superfeedr.com. The fantastic team at Van Patten Media did an amazing work to group, document and simplify all the assets used by Superfeedr. Our river.news now uses these assets by default. Also, since we use the bootstrap classes, including Bootstrap’s CSS will yield a nice looking style to your instance of river.news!

Customisable

Finally, the most recent version of river.news lets you hard-code a login and token. You can also prevent the visitor from changing the subscription list. For example, the river embedded above has disable the ability to enter a superfeedr login and password, as well as the ability to subscribe to new feeds. The configuration happens in the DOM, with data-attributes.

There’s a lot more to do! What features are important to you in an embeddable feed reader?

Instant Articles and the future of RSS

2016-03-01T00:00:00-05:00

In the past couple months, a lot has been happening on the syndication front. The mobile and social worlds have become more and more aware of the importance of the open web and we’ve seen several new efforts to improve the user experience on these fronts.

Mobile

Despite increasing bandwidth, the constraints of the mobile internet are still extremely tight: latency, spotty connections, limited memory and power… etc. The web’s constant growth when it comes to page size, number of loaded resources and the need for more and more beams and advertisement have been colliding with this.

In parallel, more than ever, we discover our content through the filter of the social networks. A couple months ago, our friends at Parsely shared that Facebook was dominating the referral traffic. The actual mileage varies from site to site, but it’s obvious that the golden days of SEO and search-engine based discovery are behind us.

Space vs. Time

Another, more massive trend is supported by these 2: the web is going toward a spatial organisation toward a time-based organization: time-lines have replaced site-maps and URLs are slowly being hidden from the user interfaces: where we consume the content is less important than what it was.

Syndication rebooted

Instant Articles are Facebook’s answer to these trends. (So is Google’s AMP). In practice, these HTML documents are embedded inside RSS feeds so that Facebook can easily import the web’s content into its apps. Like RSS, they’re not as rich as full HTML document. For example, they can’t include scripts or styles. They also require a certain number of elements which would be optional with HTML.

On their end, Facebook will replicate the content of the publisher’s site, using the Instant Article format. Each open web URL is mapped to an Instant Article inside Facebook. This is what lets them serve content much faster as they are in charge of caching and content delivery. The consequence of this approach is that once a publisher starts supporting Instant Articles, Facebook will show the Instant Article version of the content as soon as anyone shares a link to this site. You don’t have to share the links from a Facebook page : any link to the content will be served as its Instant Article version.

Some may be scared by this which results in building a shadow web, inside Facebook, but I’d like to to highlight a couple things:

This is exactly how RSS has worked since it was conceived
This is also how Google’s AMP work except that of course, in this case, Google’s cache is applied
Anyone, not just Facebook, could also cache the Instant Article version of the pages shared on their platform.

You guessed it: at Superfeedr, we’re evaluating this final and last point. Exactly like for Google’s AMP, we’re working on supporting Facebook’s Instant Article as a subscription format so that when you subscribe to a feed, you could request to only be notified with the Instant Article version of content, if it’s available.

Full Text RSS

2016-01-28T00:00:00-05:00

Of course, if your website publishes content, you should absolutely publish RSS feeds (or Atom feeds, but not both!). Luckily, a lot of publishers eventually do, even if they sometimes have trouble dealing with auto-discovery. However, a lot of these publishers will publish truncated RSS feeds, and that’s generally a bad idea.

The main reason publishers truncate their feeds is that they expect the subscribers want to visit the “original” site and then view its ads or interact with the content more… Other publishers fear that full text RSS feeds can be used by scrapers to duplicate content and rank better in search engine. Let’s face it: if somebody wants to duplicate content, there are many more ways than using the RSS feeds.

Who uses RSS feeds?

These days, it’s clear that RSS is mostly used by advanced users: millions of of them, but still a small fraction of the web’s 3+ Bn users. Given the relative complexity of most feed readers when compared to the social web applications, it’s also safe to assume that these users are at least savvy.

When it comes to ads, the most savvy web users are often the ones who use ad-blockers. Even when they don’t, they also tend to click less on these banners. So, chances are that even if they clicked in their readers to view the rest of the truncated content, they won’t even see or click on the ads.

It’s also important to remember that feed reader users chose to use a feed reader (many of them actually pay to use the feed reader of their choice). They understand their limitations as well as their benefits. It’s certainly much harder than expected to entice them to click on links away from their readers.

The feed readers are good!

Often, feed readers are offline first experiences: they aggregate content from multiple sources so that the data can be consumed while commuting, on planes… but more importantly (and more often), the feed reader users don’t have to wait for the data to load for each entry that they’re reading. Generally, feed readers will be able to load all recent stories at once, allowing for a much faster experience when reading story after story.

These days, modern feed readers also include a lot of social web features which let their users share the feed’s content, favorite the stories they like the most, and more. By doing this, the feed readers can drive a lot of traffic back to the publisher’s site.

Finally, degrading the experience can sometimes be worse than not providing any RSS feed at all. These advanced users are certainly very experienced and are also influential. If their experience interacting with the publisher’s content is great, they’re more likely to more of consume it and share it even more.

Hijacking API requests with Service Workers

2016-01-20T00:00:00-05:00

Last week, we’ve seen how to add a Service Worker to our feed reader so that it loads faster by caching its shell. One of the direct benefits of this is that our application shell is now also available offline, even when the browser is not connected to the web.

This week, we’ll see how we should also use the Service Worker API to cache the dynamic data coming from the Superfeedr Feed API.

Service Worker Toolbox

Our shell is now cached immediately after the first visit to River.news. After this, the content of the shell (static HTML, CSS and javascript) are always loaded only from the cache. (until of course we update the service worker). The RSS feeds’ content will be updated more often, so we can’t use the same approach for the calls to Superfeedr’s API. Yet, we still want the best experience so we have to cache the content and show it to the user immediately. In a way, for each call to the API, we want to achieve 2 things at concurrently:

If there is any data in the cache, show that to the user
Make a request to Superfeedr’s API and fail silently if the application is offline.

That’s a pretty common scenario in the Offline Cookbook published by Jake Archibald. As a consequence, Google implemented it in its sw-toolbox (sw-toolbox is syntactic sugar on top of Service Workers, exactly like sw-precache. The scenario we described above is called fastest.

sw-toolbox provides a routing mechanism which hijacks all HTTP requests performed by the browser and applies any of the predefined recipes in the toolbox. Here’s the code we’re using for River.news:

toolbox.router.get(/^https:\/\/push.superfeedr.com\//, toolbox.fastest);
toolbox.router.get(/^https:\/\/www.google.com\/s2\/favicons/, toolbox.fastest);
toolbox.router.get(/^https:\/\/river.news\/up.html/, toolbox.networkOnly);

We see that for any request which matches the /^https:\/\/push.superfeedr.com\//, regular expression, we apply the toolbox.fastest recipe. We use the exact same approach for requests to google.com/s2/favicons that we use to load the icons for each story.

The last line is a bit different. We only want to load this last resource using the network and never cache it.

Assessing connectivity

Caching data is extremely useful when the application only has to display information. However, if the local application needs to alter the data, caching can quickly turn into an engineering nightmare with race conditions, lost updates and more. Luckily for us, a feed reader is mostly about consuming data and rarely about altering it. The notable exception being that one might want to update their subscription list. To make things simpler, at this point, we want to disable any change to the subscription list while the application is offline.

HTML5 has its own Network Information API. Not only is it barely implemented, it also fails to detect when the device on which our application is connected to a network, but not to the web.

In order to check if we can update our subscription list, the River.news application will just make a simple HEAD request to http://river.news/up.html. If the request fails, we assume that we’re offline and then disable the form fields. If it succeeds we can let the user add or remove feeds.

The latest rule of our sw-toolbox should now make sense: we cannot afford to cache the responses if we want to reliably detect when we are offline!

There are many other use cases to Service Workers. For example, in our feed reader, we could use Service Workers to white-list a set of 3rd party javascript script.

Adding Offline Support to a feed reader with Service Workers

2016-01-12T00:00:00-05:00

Feed readers are more useful when they work offline. Unfortunately, up until a couple months ago, the web’s technologies were not compatible with this requirement. Yet, the introduction of Service Workers is slowly changing this game and, at Superfeedr, we’re excited to see open web technologies catch up with capabilities of installable native apps.

We have our very own RSS reader built on top of Superfeedr’s API: river.news. Let’s see what Service Workers can do for it and improve our code. As a reminder, this is a static single page application: there’s no application server which computes a different response for each request. Check the source code on Github.

Service Workers

Dozens of great resources online provide excellent definitions and introductions to Service Workers. To us, it’s a very significant upgrade to the web’s core philosophy. The web’s center is the HTTP protocol which stands for HyperText Transfer Protocol. Literally, this means our browsers are communicating with servers to receive hyper-text document: the web pages. Later, these documents may load Javascript code which is finally executed. With ServiceWorkers, the order changes: some javascript code can be executed before anything else happens. These workers can then hijack HTTP request (and more) and provide alternative (or fallback) responses.

Adding support to river.news would be very interesting on 2 levels:

Since the code is static, there’s no need to load it from the file server after its first load. SubToMe used AppCache for this…. but AppCache is so messy.
When loading past stories from Superfeedr, we should keep them in the cache so that the user can read them even when they go offline or have a spotty connection.

An application shell

Most web applications have a base template which is made of some HTML, CSS and Javascript used all across the application, even if there are multiple HTML pages. This application shell should absolutely be cached by the browser so that it stays in memory. There should be no latency spent downloading it because it rarely changes.

In the case of river.news, it’s actually fairly simple: there’s just a single HTML page. There’s also a single javascript file for the application’s specific code. Since we use React and Jquery as dependencies, we consider them as part of the basic shell for our application. We also use Bootstrap’s CSS, so we’ll add it to the shell, as well as a bunch of icons.

The application’s shell must immediately be cached the first time the page is loaded and we should only invalidate the cache for the files that have changed (incremental updates are much better than native apps!). Even though the Service Worker API is fairly simple, the Chrome team provides some syntactic sugar in the form of sw-precache. This script lets anyone quickly define the files required for the shell and handles invalidation without any significant code change to your own code!

That was the easy part. Next week, we’ll see how we can also cache the API calls to Superfeedr for offline usage!