powered by
LOGO
Glossary

FASTN Journal

Thoughts: Browsing History

Readership Tracking

Linking Using Package Dependency
So links never break (and original author can keep updating document URL at whim).

Git Tracking

More than one fastn package per git repo

Article Out
(on how having article.page allows us to get just the content of a page, without header/sidebar/footer etc).

fastn::Config::read()’s root should be String
15th Apr 2022

Earlier it was camino::Utf8PathBuf. Since read() is a public method, anyone calling read() must take a dependency on camino. So we changed the type of root to Option<String>.

ftd_django, the first external user of fastn::Config::read() will have String to begin with.

Doc PR. Code PR.

fastn::Config::read() now accepts optional root
14th Apr 2022

In ftd-py’s ftd_django we want the fastn package to be configurable, and stored in arbitrary named folder. So far fastn worked from a directory inside a FASTN package. In ftd_django use case, the fastn package would be in some other folder.

To handle this the code we tried was:

current_dir = os.getcwd()
os.chdir(path)
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(build())
loop.close()
os.chdir(current_dir)
This would not do as we are changing the directory while rendering. And since we are rendering from a django view, and since multiple django views could be executing in parallel in the same process, and since a process has a single global current directory, this is problematic. Even if there was only one view executing at a time this is problematic because the rest of the django code could be written with the assumption that they are called from some fixed folder, eg the django project root, and if we change the current directory that will break.

To solve this we made fastn::Config::read() accept an optional root folder. Documentation PR and Code PR.

Implementing continuous documentation for fastn
13th Apr 2022

We are now trying to implement continuous documentation for our repository. Any code change will only be merged after the documentaiton has been written and approved.

We are not yet using cdoc for this, will activate it once cdoc becomes usable internally.

Let’s see how it goes.

get-data $processor$
We just implemented a new processor called get-data. This is mainly so we can start using fastn from django templates. Checkout out get-data processor’s documentaiton.

fastn::render()
We are exposing a function fastn::render() from our fastn crate. Primary use case is our ftd Python package, which wraps fastn and ftd.

Special Packages
21st Feb 2022

We recently spoke about WASM packages. We have fastn packages that will be used to distribute static assets. We want FASTN packages for font support. We also need FASTN packages to distribute syntax definitions and syntax themes. We can maybe create FASTN package to distribute SQLite or CSV data files.

In many cases the advantage of creating FASTN package exists, even when the end users of the thing being distribuetd, font, sqlite, syntax definition, is not going to use FASTN itself.

Say you have created a font, you would want to distribute the font to both general audiance, and to people who want to use your font in a FASTN package. In such a case it would make sense for you to create FASTN package as FASTN package can become the website for you, tomorrow when we have FASTN repo in place you can easily monetise and otherwise control the distribution of your font.

Package Interfaces
Each kind of special package, will have a package interface, eg for fonts we will have fastn.dev/font, which will designate that this package is a font package, and is being used as font package. Both would be required? Not sure, explicit is better, so there is that.

FASTN cli will verify that the package is a proper font package if you say it implements: fastn.dev/font, which could mean anything, say font file must have so and so file name, or must come with fallback fonts, or it must have some meta data to be used to create CSS @font-face statement etc.

Based on the interface fastn will do something custom, font will be used in the generated HTML, syntax definition and syntax theme would be passed to syntact during syntax highlighting and so on.

Static FASTN Repo
20th Feb 2022

Once translation feature is polished, and special packages are imeplemented, processors are done, we have to start thinking about FASTN Repo. It is planned but no work has started on it. FASTN Repo is quite ambitious, we want to do a lot but how do we begin?

One option is we become target for static hosting. FASTN is going to have dynamic features, but we have not yet built them, so we can at least do static hosting. Once static fastn repo is done we can look at dynamic stuff (fastn processors running on page request time instead of publish time) and CR/online editing stuff (not sure in which order, static -> dynamic -> CR seems logical because dynamic does not have frontend UI, is purely backend work, and also maybe we will build CR/editor etc using dynamic FASTN).

Read Permissions
If we have static hosting, we can give access to read permissions. When using github pages/vercel, the permission system is pretty limited, Github Pages only allows private to people having access to some repo, which is great for an engg team, but can not be used for your entire company for example. Vercel has better story with support for multiple auth providers, like Okta, AWS Cognito, Firebase, but there is no “platform support” for authorisation rules, meaning if you are using Hugo and hosting on Vercel, it’s not easy to just say allow access to poeple from my company, you will have to do some coding.

With Static FASTN Repo we can support authorisation rules in FASTN.ftd itself. If we compare with Hugo on Vercel, it would be akin to Hugo supporting authorisation.

Hugo and Vercel are independent products, and any integration between the two requires effort. FASTN and FASTN-Repo are being developed in parallel and can have complimentary features.

Publishers Advantage
Imagine if one can say this fastn package is only readable to people in my org, meaning they have a @foo.com email address. Or to people who have followed a certain account on Twitter, or liked a certain repository on Github etc, or part of some Discord or Telegram community etc etc.

The first version of fastn-repo is being designed to realise this publisher advantage. We can start with something simple like Google Signin for company email verifiction, and then add other plugins for Slack, Telegram etc later.

Paid Packages
As a logical next step to this would be package monetisation. If you want to charge for your package, fastn-repo can help you with that as well.

Processors And Host Method
19th Feb 2022

We currently ship 3-4 processors as part of FASTN. These are part of FASTN source code, and are implemented in Rust. We also ship a few “host methods”, which FTD can call using message-host event handlers. These are implemented in JavaScript.

Having only a few such processors and methods is obviously not great because people are going to have arbitrary needs, and we can not hope to anticipate all of them and ship all such methods. We have to make it such that people can write their own.

WASM: Processor And Host Method Packages
Ideally we should be able to distribute them as FASTN packages themselves, so no one has to install cargo or npm etc. This is possible if we somehow use wasm, and ship wasm files as part of FASTN packages.

We have two alternatives, wasmer and wasmtime.

Processor Use Cases
Currently processors are written for parsing toc, which is pure computation. Then we have a processor to read file (include processor). This one needs disc access. We have a processor to do HTTP request and another to do SQLite queries against sqlite databases stored in FASTN package.

Processors And Reproducibility
If an FASTN package is pure, or depends on current package content, it is reproducible, meaning if I add that package as a dependency to some package, the package will work as expected. But if a processor allows a processor access to some resource only that machine has, say it makes an HTTP request to some intranet API, or to a database which only allows connection for given IP, and credentials, you would not want to share credentials of course. So if such a package is included as a dependency, it may not work, or it would require extra setup before it can work.

Not much we can do about it. We do not want to curtail power of processors.

Processor Sandboxing
Are there any limits to what processors can do? We will have to allow processors to read files on disc, maybe even edit them. Good thing is we have some level of control as WASM can not automatically execute arbitrary code, it can only do computation and interact with system via methods we explicitly provide.

Should we limit them to files in the package or let them read from anywhere? It becomes tricky as some use cases can exist which will need extra access. How about network access?

I think we can emulate a lot from the permission system of deno. This is good prior work, so we should try to start from there.

We can start with a very limited sandbox, only giving readonly access to files in current package.

Cloud Consideration: CPU and RAM Limits
FASTN is not going only run locally. Tomorrow we will have fastn-repo, and we may even hope github pages etc start supporting fastn-repo. So what one can do using the processors has to be designed in such a way that when fastn is not running locally but on cloud, things stay manageable.

One option is we expect every fastn-repo to run everything in an isolated server, with CPU and RAM limits imposed at docker /pod etc level.

Other option is we build resource checks in FASTN itself. It seems something like this is possible with wasmer. Ethereum implementation of wasm has support for limits. Restricting RAM should be possible based on my current understanding of WASM model (wasm runs against pre-allocated memory).

In general, having such limits is also good for FASTN running locally. We want people to have as much confidence as possible when running untrusted FASTN packages, we do not want crypto miners and other such nasties running on our end users laptops.

WASM Limitations
WASM can do pure computation, but doing arbitary thing from WASM is not possible, we have to provide access to features or functions that WASM can call to do things. But if some processor wants to make calls to arbitray C libraries etc, it’t not going to be possible in WASM model.

WASM we are chosing for making it safe to run arbitrary programs so putting limitations on what is can do is required. We can not have both, arbitrary access and safe.

Processor WASM API
For now we will only allow one method: fastn_host.get_document(id: String) -> Option<String>, this will allow a package to read a file in current package. We will have a permission “can-read-package-files”, which will have to be enabled before this will work.

FASTN.ftd
-- fastn.dependency: amitu.com/include-processor as aip
allow: can-read-package-files
implements: fastn.dev/processor
We have explicitly allowed this permission. Every package that wants to act as fastn processor has to implement fastn.dev/processor interface. We can use this processor in our FTD file now using:
-- import: aip

-- ds.code:
lang: ftd
$processor$: aip.include
$file$: hello.rs
Here we are trying to use amitu.com/include-processor, which is aliased to aip in FASTN.ftd file. By using explicit import we get to use namespacing offered by FTD.

In future we will slowly allow access to TCP, HTTP, more file operations and so on.

Host Methods and WASM
Host methods run on frontend, in Browser, after page load as part of event handlers. We can use WASM there are well.

We are going to have a “rich stdlib”, we do not want to create meaningless fragmentation. We believe in batteries included. We have the luxury of only including the code of a host method if that method is being called from the current page. So a big library is okay.

Host methods will be able to call the “fastn stdlib”, it would be tricky to detect what libraries have been called tho, one can not do source code analysis, source is already compiled into wasm. Maybe we can analyse wasm to see all calls to outside world. But for the purpose of this discussion lets assume it is possible (if not we will have to create some sort of opt in, a method can only call stdlib functions it has declred in some form of manifest file, sucks, but what can we do, we can not include entire stdlib in all pages generated using FASTN after all).

host-method package dependency
-- fastn.dependency: amitu.com/parse-date
implements: fastn.dev/host-method
Any package that provides a host method shoud implement fastn.dev/host-methods package interface.
-- ftd.input:
$on-change$: message-host: amitu.com/parse-date.parse, error=error-variable-name
> target=where-to-store-date
Say want to parse the date entered by the user in an input field, and we have a package amitu.com/parse-date, which provides a host method “parse” that can do this.

Where Would WASM Files Be Stored?
We can store wasm files in FASTN folder. Maybe we can have each wasm file export a single function, so we only include the minimum amount of code. For processors we can club all of them together in one file.

Should Processors Be Callable From Frontend?
Currently the backend processors only work with data, from FTD’s perspective. FTD contains a bunch of variables, and processors create mode data that gets added to FTD variable space. Frontend methods also interact with FTD via variables.

The capabilities available to frontend does not make sense for backend. But backend one may make sense for frontend. Though not backend only functionality is used, like file system access or database query, network etc.

There can be some overlap, some functionalities should be available both in frontend and backend. But it can not be generalised.

Static Assets From Dependencies
9th Feb 20222

For static assets we have to solve three problems.

Where are the assets stored?
In the main package, we create a .build folder. The dependencies are checked out in .packages folder.

For a dependency d.com/a, the asset images/foo.png will be stored in .build/_assets/d.com/a/images/foo.png.

There is not much to this part.

When to copy?
Should we copy all assets in .packages folder in .build/_assets? Or should we do it on demand?

Easy answer for now is lets copy all static (non ftd and md files a package are considered static). In future we can optimize this.

How to ensure paths work?
This is the trikiest part.

So say d.com/a/lib.ftd has a component:

-- ftd.image logo:
src: $assets.files.static.images.foo.png
We are going to use relative path here. Technically if someone uses absolute path, then we can still learn if the absolute path is part of package or outside, e.g. if they used:
-- ftd.image logo:
src: $assets.files.static.images.foo.png
This is the worst case scenario. They should not use it as if they do, when they are developing logically they would not see the local modifications to foo.png without publishing.

But technically we can rewrite these URLs too. We do not want to continue to download from d.com, and increase their hosting bills, and also take the risk of them blocking assets via some tweak in their nginx conf etc, leading to our site giving 404.

So we should rewrite the URL to: /_assets/d.com/a/images/foo.png and now it would be server by our hosting providers and things will be in our control.

Repeating, technically we can rewrite such URLs as well, and maybe we should. Then comes next hard part:

d.com/a/lib.ftd
-- ftd.image logo:
ftd.image-src src: $assets.files.static.images.foo.png
src: $src
We have to change the src again to /_assets/d.com/a/images/foo.png. Should we rewrite all paths? What if this was part of the main package? Or what if we did this from our main package:
-- import: d.com/a/lib

-- lib.logo:
src: $assets.files.static.images.foo.png
Now is this referring to $assets.files.static.images.foo.png from our package, or $assets.files.static.images.foo.png?

We can argue we will only handle relative URLs and not absolute ones. Further when we come across a relative URL we will resolve it to full URL based on current package. We already track the current package, so this should be easy.

So same string in different package will refer to different values. This becomes more complex in this scenario:

d.com/a/lib.ftd
-- string logo-1: images/foo.png
-- string logo-2: images/two.png

-- ftd.image logo:
ftd.image-src src: $assets.files.static.images.foo.png
src: $src
main package:
-- import: d.com/a/lib

-- lib.logo:
src: $lib.logo-2

-- lib.logo:
src: $assets.files.static.images.two.png
Here both values are same, but should they resolve into same URL or different URLs?

What if we say neither URLs are right and the lib should include path including package name:

d.com/a/lib.ftd
-- string logo-1: $assets.files.static.images.foo.png
-- string logo-2: $assets.files.static.images.two.png

-- ftd.image logo:
ftd.image src: $logo-1
src: $src
main package:
-- import: d.com/a/lib

-- lib.logo:
src: $lib.logo-2

-- lib.logo:
src: $assets.files.static.images.two.png
In this we are never having to worry if images/foo.png refers to amitu.com or d.com/a. It’s explicit.

It is still a “relative url” because it does not start with / or protocol.

So we look at all relative URLs containing package name, and either remove the package name: amitu.com/images/two.png -> images/two.png or add _assets: d.com/a/images/two.png -> _assets/d.com/a/images/two.png. We can even resolve https://d.com/a/images/two.png -> _assets/d.com/a/images/two.png.

If the relagive URL doesnt contain current package or dependency name then we leave it as is.

Theme designers would be encouraged to use their package name in the assets.

Base And Versioning
Instead of resolving https://d.com/a/images/two.png -> _assets/d.com/a/images/two.png etc we can do https://d.com/a/images/two.png -> $HOST$BASE/_assets/d.com/a/images/two.png when --base argument is used. Where $HOST would be either http://127.0.0.1:8000 etc or https://amitu.com. This is to ensure image URLs not depend on BASE. This way we can use $BASE for handling markdown URLs.

Markdown URL problem: tutorial/ should resolve into $BASE/tutorial, this works if base is common for the entire project as has been so far. But when we have versioning, we may want tutorial/ to resolve either into /v1/tutorial/ or /v2/tutorial/, and this will depenend on current version. So when we have haversion support we will want to keep base free to handle versioning, but we are assuming assets would be shared across all versions.

On main:
4th Feb 2022

So Shobhit has implemented the “main” proposal. What we are trying to avoid is imports at top of each file. With main we manage that. But now we lose discoverablity, where does a symbol come from. In main proposal the fact author is coming from foo.com/book-theme is not obvious as you are writing:

FASTN.dev
-- fastn.dependency: foo.com/book-theme
implements: fastn.dev/blog
fastn.dev/blog declairs main: book, author which tells you that because of the implements every module in current package has access to author.

Revised proposal
The idea of main: was to ensure we do not have all the code in index file, so we can then have auto-import feature:

FASTN.dev
-- fastn.dependency: foo.com/book-theme
auto-import: foo.com/book-theme/author
auto-import: foo.com/book-theme/article as ft
With this author would be available to every module in current package.

Further we will not have : (reason: when using / we get a full URL that we can paste in browser and see doc of individual module) and we will always refer to current package either as package keyword, or we can even allow as alias syntax in the -- fastn.package: line.

Static Assets From Dependencies
Lets say we let any package specify some exports, these could be images or even ftd files (eg contact-us, about-us can be auto created without cluttering).

theme/FASTN.ftd
-- import: fastn
-- fastn.package: theme
-- fastn.export: images/
default: true
-- fastn.export: sample/about-us.ftd as about-us.ftd
default: false
When I am using the theme I can be explicit:
-- fastn.dependency: theme
include: *
include: * would be default. We have can have include: defaults to only import default ones. We can also include individual patterns explicitly.

Note: exports are not considered part of package interface.

Can we import any file?
Not seeing any reason why not. There is a question of backward compatibility, I as a theme author may want to move things around, things that are not explicitly exported are private to me.

This feature is largely so components exposed by a theme can reliably refer to assets made available by theme. Ideally only theme components should know about existence of such.

Auto Include In A Name-spaced Folder
In .build we can create .build/foo.com/blog-theme/images/bg.png etc, and we can do it on demand (only if any component that refers to bg.png is actually used while rendering any document it would be copied). Since assets in this scheme can never conflict, they can considered opaque, so the package can not refer to them.

Problem would be say a theme has a few images and some component in that theme lets you pass one of the those images as component argument: so how would we refer to that image? Full URL or the partial URL? We can refer partial URL and since component knows its part of some theme, it will resolve. But what if in author package there is some other image and I do not want theme provided URL and use my URL? We are then forced to give full URL and not partial. A component in a theme refers to images in the theme via relative URL, eg images/bg.png. One option is we always use full URL: foo.com/book-theme/images/bg.png. At least in this case, because URL starts with package name, we know it belongs to that package, and so we can know if its either https://foo.com/book-theme/images/bg.png or https://author.com/FASTN/foo.com/book-theme/images/bg.png. If the theme component referred to the URL using protocol, then it would be absolute and we do not want to analyze.

Ideally we should copy on access.

Package Notifications
3rd Feb 2022

I was going through MagicBell and I quite love them. It gave me the idea that people today almost universally hate the email subscription dialog, and yet universally are driven to clicking that red icon.

Imagine any FASTN page shows a notifications icon like this in the header. We do not have login but we have cookie, we keep track of what you have read via cookie or local store. In the future we can also include signup via email option as in the notification drop down. Even subscribe via browser using Push API.

Email and Push will require server side support, but basic notification without that is also quite useful and can be easily implemented: if we somehow solve the how in FASTN package the notifications are stored problem.

What to show in drop down and link for each of them has to discoverable by fastn somehow. Maybe we can have:

fastn.ftd notification variable
-- record news-item:
caption title:
string link:
optional ftd.image-src src:
body description:
datetime created-on:

-- news-item list news:
So people can create notifications in say config.ftd:
-- import: fastn

-- fastn.news: Package Notifications
link: /journal/#package-notification
created-on: 2022-02-03 11:30AM IST

A spec for showing notifications in header has been just added to Journal.
We will ask people to keep only some meaningful number of items in news, say 10, and keep deleting the old ones. Or we can keep the entire archive and only consider the first 10 when showing the notifications. We can also show all the news items in /FASTN/news/ page.

We will create a package: fifthtry.github.io/notifications which will act as the widget, read the fastn.news and show. We will have a message-host function to store the latest notification created-on in a cookie, and on page load the value of the cookie would be made available on fastn.latest-news-read variable, so any news items later than fastn.latest-news-read will be shown as unread.

We will mark the news as read when the notification icon is clicked.

To impelment this we need date-time type ideally. Else if we store created-on as string, we still need > check. We also need a way to get first item from a list. To show the dot on the closed icon state, we need check of cookie value vs created-on of latest news. created-on of latest news can be stored as a another fastn variable for convenience till we get first item of a list support. Once the UI is open, we will want to show what all is new, so we will def need string comparison operator, >.

FASTN Package Registry
Currently FASTN is fully distributed, each FASTN package is published on a URL controlled by the owner of the package. To aid discoverablity tt would be cool if there was a package registry though, a central place where you can quickly discover packages.

Is this going against our decentral design? Not really. FASTN does not really depend on the registry unlike other package managers. You can depend on any package and if some package is removed from package registry, all it will lose is a listing one some site. The package itself would be still available and anyone who want to use the package can still use it.

Registry is just a handy place to quickly find themes, components etc, we can easy show all (most) packages that implement a given package interface. We can even show some stats like how many packages depend on any given package, which can be a good proxy for people who want to create a dependency on that package, a package that is quite popular for example maybe safer to use than a package that just came up yesterday and no one is using it.

We will make fastn.dev the “fastn central” or “fastn package repository”, which means eventually fastn.dev will have dynamic stuff. Can we build the dynamic stuff using our proposed FASTN Web Framework? Time will tell. We will not block on it. FASTN package repository can tomorrow become an open source real world showcase of fastn web framework when it is in place tho.

fastn ping
The way the registry would work is by a package pinging the central repository that the package has been updated. The ping itself will not be trusted, we will not have API keys, signup etc. fastn ping will simply call an end point and pass the package name as the only information. FASTN registry will then schedule a crawl of the fastn package and get the information from the published package.

Registry and News
We just discussed adding news support in FASTN packages. FASTN registry can show the news, and may give quick notify me when this package has news button next to each package listed.

Registry And Web Hook
Say you are authoring an FASTN package and it depends a few FASTN packages, it would be cool to auto re-publish your fastn package every time any of the dependencies changes.

Since fastn ping is notifying the registry when package changes, fastn registry can tomorrow also notify arbitrary webhook end points when a news is published about any package. You can listen for that and review and republish or auto-republish.

Decentralized Registry?
The first idea was a centralised registry, but we can have say 10s of registries each gossiping with each other telling about updates, so there is no central point. And any registry you publish to, you can expect the eventually the entire registry network will know about updates to your package.

More on package.zip
2nd Feb 2022

THe idea is we will create package.zip in .build folder whenever we do fastn build. Currently the zip file we download is generated by Github and contains the source of entire repo.

In the package.zip we will include the entire repo.

- FASTN.ftd
- src
  - index.ftd
- code.rs
In this case the repo has a single package, and the build will simply include all the files of the package minus the files ignored by fastn build (we auto ignore some files like .git etc, and we allow FASTN.ftd to specify other files and patters to ignore).

Consider a repo with multiple FASTN packages:

- en/
  - FASTN.ftd
  - index.ftd
  - a.ftd
  - image.png
- hi/
  - FASTN.ftd
- code.rs
In this case when we are are calling fastn build we have to cd into one of the two folders, and each en and hi will get their own .build folders containging their own package.zip files.

The zip will still include all the files in the repo based on previous ignore logic. We will introduce a new ignore heuristic, if you come across a file that belongs to another FASTN package, auto ignore it as well. So when building package.zip for en folder, code.rs will be included but not hi folder.

A package can opt out of auto-ignore-other-packages-in-zip heuristic by overwriting this key to false in FASTN.ftd. This heuristic is only to reduce the package.zip size.

The package.zip will also contain FASTN.manifest.ftd file which will be currently contain only one setting: the location of FASTN.ftd file that should be the starting point. Since if we have disabled auto-ignore-other-packages-in-zip heuristic, en/.build/package.zip will contain both en and hi folders, and without the manifest file we do not know where the FASTN.ftd file is so we need the manifest file.

Confusion On Ignore
Currently we use “ignored files” terminology to refer to both files that never want to know about, eg .git files, vs the files we know about but currently we want to not copy them to .build folder, eg .history or .track folders.

Now onwards “ignored files” terminology means files that ignored from all fastn commands, they are as good as they are not on disc.

Among the non ignore files, there are files like .history folder files which some fastn commands need but not others.

Currently we assumed ignored means files that fastn build ignores, but now we want to say ignored files are files that entire fastn system ignores, and non ignored files but files which are still ignored by fastn build has no global name, its fastn build’s internal detail, and fastn build handle it how so ever is effecient.

code.rs is part of package?
Say we have this folder structure:

- en/
  - FASTN.ftd
  - index.ftd
  - a.ftd
  - image.png
- hi/
  - FASTN.ftd
- code.rs
When someone imports en package, they can import en:a.ftd, but can they also refer code.rs directly somehow?

Decision: no, code.rs is part of package.zip, but is not considered part of en package. Only files directly insider en, or in case hi files in hi or in en (as part of our fallback logic) are considered part of the package.

So you can import hi:a, even though there is no such file because we have fallback logic.

Bind Proposal
One option is we do not let people go out of en folder, and if any file has to live outside, that file can only be accessed after it has been explictily bound In FASTN.ftd:

-- import: fastn
-- fastn.package: en
-- fastn.bind: ../code.rs as foo.rs
-- fastn.bind: ../images
This is like we are implementing our own file linking feature.

Solution 1
With bind the generated package will simply copy the bound folder. So we will not care of repo structure at all when generating package.zip. When bulding package.zip for en, we will start from en folder, the zip will contain all content of en folder, and then all bind files/dirs. So we will do cp ../code.rs foo.rs and cp -r ../images .

Solution: Final?