/* This file is part of Bolixo. Bolixo is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. Bolixo is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with Bolixo. If not, see . */ #include #include #include "../bolixo.m" #define EXTERN #include "techdoc.h" #include "steps.h" void document_webapi_proto(); static void blackhole_link() { document_clickable_link ("https://solucorp.bolixo.org/projects/blackhole","blackhole"); } void bolixo_techdoc (DOCUMENT_POINTER &docpointer, DOC_ID &jump, const char *username, bool display_full, bool index) { glocal bool unknown_user = false; glocal const char *username = username; if (username[0] == '\0'){ glocal.unknown_user = true; glocal.username = "some_user"; } htmlout ("\n"); ("",docpointer,jump,display_full,index); <- section("Distributed system");> The Bolixo network is made of independant servers. There is no central server storing users data. Servers interact directly with each other. Further, in Bolixo, there is no single namespace for user accounts. Each server has his own user accounts. Server A may have accounts Jack, John and Mary. Server B may have the same account names. But Jack@A is not jack@B. <- section("Servers");> Normally, a single server (a single VM) is all that is needed to run Bolixo. The server is associated with a domain name (alpha.bolixo.org for example). Servers communicate with other servers over HTTPS.

There are two server types:

Public
A public server allows anyone to create an account. There is no implied relation between two user accounts on a public server.

Private
Private servers have two main differences.
  • Only the administrator (sysadmin) may create accounts.
  • Users may interact with each other without having to establish a relation first. They are members of the same organisation or company. As such, they are expected to communicate together.
But except for those differences, users on a private server may interact normally with users on public servers.

While Bolixo services are normally insalled on a single physical server, it is possible to spread the services on multiple servers. This is all transparent for the services. The /- blackhole_link(); system connects all those services together, on whatever physical server they run. A simple configuration file tells where each service is run and from there, the blackhole configuration is produced. <- section("Digital signature");> All content created by users is digitally signed. Every user has a personnal 2048 bits key. Content is signed using the private key and other servers may validate the content using the user public key. The micro-service /- tlmpdoc_link(section_keysd,"keysd"); manage all the user keys.

When messages are copied from servers to server, the digital signature follows. A public API allows any server to retrieve the public key of a user. This is done the first time a server receives some content from a remote user. The public key is then recorded. It will be used to validate any future messages. <- section("Remote accounts",section_remote_accounts);> The remote account concept is a key to allow the bolixo system to scale. A remote account has the following caracteristics:

It is fully qualified
Account jack on server A is represented has jack@A on any other server. So jack@A on server B does not conflict with local user jack.

It has no password
It is not possible to perform a normal login on this remote account.

Login only accepted from origin server
A server may perform a remote login on another server, on behalf ot a given user using his private key. The sequence goes like this.
  • Server A attempt to perform a remote login on server B for user jack.
  • Server B replied with a challenge (a small string). Server B never generate the same challenge.
  • Server A will sign this challenge using user jack private key.
  • Server B will validate the signature using user jack public key.
  • A session ID is generated by server B. Server A can now perform pretty much any action a normal user can do on server B.

They are created on the fly
Remote accounts are created as needed, whenever a server receives a message from another server.

Remote accounts allow user content to be copied to other servers. One important side effect is that most user operations on a server are local. When a user lookups his news feed, it is a local activity. <- section("Messages");> <- section("Messages folders");> <- section("Messages propagation");> <- section("Public messages");> <- section("Private messages");> <- section("Group messages");> <- section("Projects");> <- section("bolixo.org directory");> The bolixo network is made of independant servers. Each server has its own users. How do we locate friends in the network ? The server /- document_clickable_link("https://bolixo.org","https://bolixo.org"); is the central repository used to search and locate user accounts. Bolixo.org is not required to run the network. It is only useful to find other users in the network.

Bolixo.org is avaibale to anyone
No need to have a Bolixo account to use the server

Account creation
Servers may register to bolixo.org. They register their capacity (maximum number of accounts) and they record the email of their end users. To create an account, the user just visit Bolixo.org. It finds an available server (with suitable capacity) and sends the end user there so it can complete his account creation.

Publish to Bolixo.org
By default, users are invisibles. The only way to reach them is by knowning their user account and Bolixo server name. But users may publish different informations to Bolixo.org (even their phone number if they want). There is a form in the account configuration just for that.

Once you have found another user, you do not need Bolixo.org to interact with him.

Locate a server
If a user has forgotten his assigned server, he may uses Bolixo.org to perform the first half of the login (get user email) and then jump to the user server to complete the second half (password).

Webapi
Bolixo.org provides some REST apis. This allows bolixo servers to query its database.

Read Bolixo documentation
The complete documentation (end user and technical) is available from Bolixo.org.

<- section("Architecture");> Bolixo is made of components running in LXC containers. The containers are called LXC0s. Here is a diagram presenting the relation between Bolixo components. In the diagram, rectangles represent a LXC0 container. Ellipses represent a program. In general, containers are running a single program. To simplify the view containers running a single program are just rectangles (no ellipse inside).

/- document_clickable_img ("/bolixo-arch.jpg","40%",1);
<- section("LXC0: Micro-containers",section_lxc0);> LXC0 are standard LXC containers (Bolixo is running a standard Linux kernel). The 0 stands for its minimalist approach. A LXC0 contains only the necessary files used to provide the service. It usually runs a single program plus a tiny init. It contains the programs, the libraries needed and few resource files. It does not contain any packages, logging utilities, sshd. It does not contain a shell. Here is what the ps axu command presents about the writed component.

<- pre();> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 3121992 0.0 0.0 5608 1644 ? Ss 21:00 0:00 /sbin/init bolixo 3121997 0.0 0.0 21908 7012 ? S 21:00 0:01 /usr/sbin/bo-writed

The writed container contains 44 files. A LXC0 container is assembled on the fly at startup. The recipy to assemble a LXC0 container is created using the strace utility. You run the target program outside a container under strace control. A utility analyses the log produced by strace and extract the precise list of files needed to run the program.

This was done to create the most precise container possible. If there is a breach in one container, the attacker will end up in an unfriendly place. It push the concept of server hardening to a new limit.

All containers in Bolixo are lXC0s. This includes containers running Apache, Mariadb and exim. <- section("Components");> Components are presented in alphabetic order.

<- section("Communication between components");> By default, Bolixo components (containers) can't use IP. They can't talk to each other directly. Although each LXC0 is assigned an IP number (192.168.122.X), iptables rules prevent connection between members of the 192.168.122 network. They usually lack an IP gateway. They can't reach the Internet and the Internet can't connect to them.

Bolixo components usually use unix socket to communicate. The /- blackhole_link(); installs some unix sockets in a component at startup time. For example, in every component, blackhole installs the unix socket /dev/log, allowing the component to log messages. Bolixo has its own logging system, running outside components. The blackhole have connection rules for each component and will establish the proper connection based on the source component and the unix socket used. <- section("bod: the reader");> Bod is used to get information, read messages, read files etc... Bod has read access to the bosqldata database. It has also read only access to the directory /var/lib/bolixo.

Bod is also the gateway to writed. No other component can reach writed.

Bod may use IP to reach the Internet and other Bolixo servers, using the webapi protocol. It manages messages and configuration synchronisation with other Bolixo servers. <- section("documentd: games and documents");> Documentd handles games and documents. It receives documents and games to edit from the bod component. It receives command to apply to documents and return the new state of the document. Documentd has only access to temporary storage. This is used when restarting the service to save current documents and games. <- section("exim: mail server");> Exim is used to send various notifications to end users. <- section("keysd: private and public key management",section_keysd);> Each Bolixo user has a 2048 bits private key. This key is used to sign messages created by the user. It is also used for /- tlmpdoc_link (section_remote_accounts,"remote login."); Also, every Bolixo server has a 2048 bits private key used for communication/authentification between Bolixo servers.

The private keys are located in the /- tlmpdoc_link(section_bosqlduser,"user database."); The keysd component generates the keys and make use of them. The private keys are pass phrase protected. Only keysd knows the passphrase. The private key never moves out of keysd. <- section("protocheck: web protocol checker",section_protocheck);> Protocheck is a web protocol checker, part of the blackhole system. It does a minimal validation and can terminate the communication if it detects an anomaly. <- section("publishd: push information to other Bolixo servers");> Publishd receives messages from writed and copy those messages to relevant Bolixo servers. It is used for public messages and for group messages. When one Bolixo user express interest in the public message of another Bolixo user, located on a different server, a remote interest request is made. When user A on server1 expresses interest in user B on server2, a request is made. When user C on server1 does the same, no request is done. Said differently publishd on server2 will copy a message once for all users on server1. <- section("sessiond: session manager");> The session manager records information about the active sessions of users. It records mostly UI information such as the tab states, states of dialogs, notifications. <- section("web and web-fail: web content");> The web component runs four services:

  • index: This is driving the Bolixo user interface.
  • public: This is presenting public pages.
  • webapi: This is converting the Bolixo web protocol into Bolixo internal protocol.
  • bo-websocket: This is handling websocket connections.

The web and web-fail pair are used for failover when services are restarted. See the /- tlmpdoc_link(section_bolixo_production_restart,"bolixo-production restart"); command for a better understanding of the web-fail and webssl-fail components. <- section("webssl and webssl-fail: ssl processing and static content");> Webssl handles SSL certificates and static content (mostly images for documentation and UI). <- section("writed: the writer");> Writed is the component updating databases. It updates the user database and the content database. <- section("Blackhole");> Blackhole is the central component connecting each Bolixo services to each other. More information about Blackhole is here: /- document_clickable_link("https://solucorp.bolixo.org/projects/blackhole","https://solucorp.bolixo.org/projects/blackhole");

Blackhole provides the following functions:

Routing agent
As explained in section /- tlmpdoc_link(section_lxc0,"lxc0"); , Bolixo service can't talk to each other directly. In general, the service can only use unix domaine sockets inside its LXC container. The sockets are created by the Blackhole. Using connection rules, the blackhole routes requests done by a service to another service. A connection rules has 2 parts.

  • Source: server container ip/unix port
  • Destination: server container port

Load balancer
If there are multiple rules with the same source, but different destinations, Blackhole acts as a load balancer

Protocol validation
Connection rules may introduce a protocol checker. For HTTP request, the protocheck micro-service is used. See /- tlmpdoc_link(section_protocheck,"protocheck"); in this documentation.
<- section("Databases");> There are two database servers in Bolixo, each running in its own LXC0 container. MariaDB is used in both cases. The Bolixo.org runs another database.

<- section("bosqlduser: user information database",section_bosqlduser);> <- section("bosqlddata: content database",section_bosqlddata);> <- section("Bolixo monitor",section_bo_mon);> <- section("Bolixo logging system");> <- section("Programming language");> <- section("Instrumentation",section_instrumentation);> <- section("File system");> <- section("Web API");> This the web api. /- document_webapi_proto(); <- section("Operations");> This document is not a Bolixo administration guide. But the information provided here is useful to understand how Bolixo works.

<- section("bolixo-production",section_bolixo_production);> Operations on Bolixo components are done using the bolixo-production command line tool. This is a long name, so a shorter name is installed: just bo. The bolixo-production (and bo) command supports tab completion. Further, just typing the bo command without argument produces a help screen. There are many sections. We will see the main commands here.

<- section("Production");> You will use mostly those commands. <- section("status");> Using the following command, you can get a summary of the state of one component. The bo status without argument prints the list of available Bolixo components. <- pre();> bo status documentd You get the following output for the documentd component. This information is useful for developper. Instrumentation may be turned on and off for any components. See the section /- tlmpdoc_link(section_instrumentation,"Instrumentation"); for more details. <- pre();> Version SVN_4645 instrument: disable protocolstats: 0 chessmaxskill: 3 bytes sent: requests=0 content=0 script=0 unotify=0 notify=0 nbwaiting: 0 handles: 0 subprograms: 1 subprogram CHES: nbsend=4 nbrec=26 gameid= command=/usr/bin/stockfish <- section("test-system");> This produces a complete report of the Bolixo components. Components with an issue are starting with an arrow ->. Here is a sample report. Most components have a test API. Components are free to implement the test request and provide all kind of results. In general, a component will perform a test request on all the other components it talks to. The /- tlmpdoc_link(section_bo_mon,"Bolixo monitor"); runs these tests every 30 seconds. On the line starting with a *, we see the bod component test. Internal_error1=0 means communication with bod was successful. Writed=1 means the test request sent to writed was successful.

There are many lines for the bod component because several bod processes are running and they are all tested. The blackhole is load balancing bod requests. But in this case, the blackhole system has one test rule per bod process. Theses rules are only usable by the monitor system. This bypasses the load balancer. <- pre();> success = 0 SwapFree ok=1 freekb=245424 loadavg ok=1 0.18 0.30 0.48 2/6028 1272784 diskfree ok=1 f_blocks=120203246 f_bfree=23965689 space=91.421848 /var/run/tests/A-keysd-localhost.sock: internal_error=0 /var/run/tests/A-sessiond-localhost-admin-9200.sock: internal_error=0 success=1 * /var/run/tests/B-bod-localhost-admin-9000.sock: internal_error1=0 writed=1 bdfiles1=1 bdfiles2=1 bdusers=1 sessiond1=1 sessiond2=1 keysd=1 fsok=1 publish_dbfiles=1 publish_fsok=1 docd=1 adm_sess=2 /var/run/tests/B-bod-localhost-admin-9001.sock: internal_error1=0 writed=1 bdfiles1=1 bdfiles2=1 bdusers=1 sessiond1=1 sessiond2=1 keysd=1 fsok=1 publish_dbfiles=1 publish_fsok=1 docd=1 adm_sess=2 /var/run/tests/B-bod-localhost-admin-9002.sock: internal_error1=0 writed=1 bdfiles1=1 bdfiles2=1 bdusers=1 sessiond1=1 sessiond2=1 keysd=1 fsok=1 publish_dbfiles=1 publish_fsok=1 docd=1 adm_sess=2 /var/run/tests/B-bod-localhost-admin-9003.sock: internal_error1=0 writed=1 bdfiles1=1 bdfiles2=1 bdusers=1 sessiond1=1 sessiond2=1 keysd=1 fsok=1 publish_dbfiles=1 publish_fsok=1 docd=1 adm_sess=2 /var/run/tests/B-bod-localhost-admin-9004.sock: internal_error1=0 writed=1 bdfiles1=1 bdfiles2=1 bdusers=1 sessiond1=1 sessiond2=1 keysd=1 fsok=1 publish_dbfiles=1 publish_fsok=1 docd=1 adm_sess=2 /var/run/tests/B-bod-localhost-admin-9005.sock: internal_error1=0 writed=1 bdfiles1=1 bdfiles2=1 bdusers=1 sessiond1=1 sessiond2=1 keysd=1 fsok=1 publish_dbfiles=1 publish_fsok=1 docd=1 adm_sess=2 /var/run/tests/B-bod-localhost-admin-9006.sock: internal_error1=0 writed=1 bdfiles1=1 bdfiles2=1 bdusers=1 sessiond1=1 sessiond2=1 keysd=1 fsok=1 publish_dbfiles=1 publish_fsok=1 docd=1 adm_sess=2 /var/run/tests/B-bod-localhost-admin-9007.sock: internal_error1=0 writed=1 bdfiles1=1 bdfiles2=1 bdusers=1 sessiond1=1 sessiond2=1 keysd=1 fsok=1 publish_dbfiles=1 publish_fsok=1 docd=1 adm_sess=2 /var/run/tests/B-bod-localhost-admin-9008.sock: internal_error1=0 writed=1 bdfiles1=1 bdfiles2=1 bdusers=1 sessiond1=1 sessiond2=1 keysd=1 fsok=1 publish_dbfiles=1 publish_fsok=1 docd=1 adm_sess=2 /var/run/tests/B-bod-localhost-admin-9009.sock: internal_error1=0 writed=1 bdfiles1=1 bdfiles2=1 bdusers=1 sessiond1=1 sessiond2=1 keysd=1 fsok=1 publish_dbfiles=1 publish_fsok=1 docd=1 adm_sess=2 /var/run/tests/B-bod-localhost-admin-9010.sock: internal_error1=0 writed=1 bdfiles1=1 bdfiles2=1 bdusers=1 sessiond1=1 sessiond2=1 keysd=1 fsok=1 publish_dbfiles=1 publish_fsok=1 docd=1 adm_sess=2 /var/run/tests/B-documentd-localhost-client.sock: internal_error=0 success=1 /var/run/tests/B-publishd-localhost-client.sock: internal_error=0 dbfiles=1 fsok=1 /var/run/tests/B-udpproxy-out.sock: test ok, nbreq=247021 /var/run/tests/B-udpproxy.sock: test ok, nbreq=247021 /var/run/tests/C-exim-localhost.sock: error=0 220 bolixo.org ESMTP Exim 4.98 Thu, 21 Aug 2025 20:08:40 -0400 /var/run/tests/D-web-fail-stop-localhost.sock: run=1 /var/run/tests/D-web-stop-localhost.sock: run=1 -> /var/run/tests/E-syslog-localhost.sock: errors=1 last=2025/08/20-19:53:55 /var/run/tests/F-web-80-localhost.sock: ok seen /var/run/tests/F-web-fail-80-localhost.sock: ok seen /var/run/tests/F-websocket-fail-localhost.sock: test ok, workers=0, paused=0 /var/run/tests/F-websocket-localhost.sock: test ok, workers=0, paused=0 <- section("restart",section_bolixo_production_restart); > This is the way you restart Bolixo components. Some sub-commands are used to restart a group of components (such as most explained below). All components may be restarted at any time without much impact for end users. There are two solutions to achieve this: one is managed by the web component and the other, allows you to restart the web component.

The solution to restart most components in Bolixo lies in the web component. It is a gate system called trli-stop. Whenever a request is received in the web component (by index, public, or webapi), a permission request is sent to trli-stop. trli-stop simply replies yes. Once the component receives its yes confirmation, it continue processing the web request. The connection to trli-stop remains open until the request is complete. When we need to restart some bolixo components, we tell trli-stop to stop. Immediatly, trli-stop stops replying to permission requests. So all new activities in the web components will pile at the gate. trli-stop knows about all the other requests it has granted permission (yes) to complete, but still in process (in flight). trli-stop waits until all in flight processes are completed (the connections to trli-stop are dropped). At this point, trli-stop knows that all internal Bolixo components are idle. They may be restarted without problems.

The bo-websocket service inside the web component has its own way to handle the gating. Bolixo-production uses both trli-stop and bo-websocket to ensure the internal components are idle. So to restart the writed component, you just type: <- pre()> bo restart writed Here are the list of sub-commands used with restart:

internals
Restart everything except the databases, webs and exim

most
This is the sub-command you will use after a bolixo update. It restarts internals and webs

sqls
Restart all databases. You seldom need to do that.

bod, documentd, protocheck, publishd, sessiond, writed
Those components may be restarted separatly. Other components will reconnect on the fly. Restarting those components may be done any time. You seldom need to restart those separately.

keysd
Keysd manage the user private keys. You can restart this component like any other, except for one important thing. Keysd requires a passphrase to decrypt the 2048 bits keys read from the database. When you restarti keysd, you must provide the passphrase. There are two ways to provide the passphrase:
  • You can write it down in the file /root/keysd.pass. The file /root/keysd.pass will be securely deleted (using the shred command) during the restart process.
  • You provide it interactivly.

webs
When we restart web and webssl, the following steps are executed.

  • Normally, web and webssl are receiving all requests. web-fail and webssl-fail are idle.
  • Web-fail and webssl-fail are restarted.
  • Priority is switched from web,webssl to web-fail,webssl-fail.
  • We wait until all connections have ended on web,webssl.
  • We restart web and webssl.
  • We switch back priority back to web and webssl.

The /- blackhole_link(); system acts as a load balancer for the web,webssl and web-fail/webssl-fail services.

Using this strategy, we can upgrade/restart the web and webssl services at any time without disrupting user activities.

<- section ("update-script"); > When installing a new version of the Bolixo package, there is no immediate effects on the running components: they are running in LXC0 containers, so are isolated from the rest of the system. But before running the command: <- pre();> bo restart most to activate the new version, some tasks may have to be done. For example, some change must be done to the database model to support new feature. How do you know what must be done before running the new version ? This is what update-script is about. The Bolixo package deliver the file
/usr/share/bolixo/update-script
. This file contains all the updates needed for all Bolixo version. The file looks like this: <- pre();> ----recipients-move-dirs_content recipients field moved from table files to dirs_content bo calltest createdb-patch9 ----onlylocal new option for bod bo config Lines starting with ---- are keys. Only the first word after ---- is the key. The rest of the line is a comment. Other lines are commands needed to update the system. The database /- tlmpdoc_link(section_bosqlduser,"bosqlduser"); contains a table named updates. The table contains the keys of all updates already applied. The update-script command will find the missing updates, apply them and update the database.

In the sample above, we see that for the update named recipients-move-dirs_content, the command <- pre();> bo calltest createdb-patch9 muse be executed. For te update onlylocal, the command <- pre();> bo config must be executed. See the section /- tlmpdoc_link(section_bo_config,"config"); for more details about this command.

There are two options for the command update-script. The first simply test and report what must be done. The second executes it and update the database. <- pre();> bo update-script --test bo update-script --doit A typical update sequence for Bolixo goes like this: <- pre();> Update the bolixo package bo update-script --doit Copy the passphrase for keysd in /root/keysd.pass bo restart most <- section ("Syslog");> Bolixo has its own logging system. You can access it using some bo commands. Here they are.

syslog-clear
Clear all messages in syslog.

syslog-logerrs
Shows only error lines.

syslog-logs
Shows all lines: errors and warnings

syslog-reset
Reset errors in syslog. The error messages are kept, but the status screen report zero errors. This is used by the monitoring system. You generally use this when you have reviewed the errors.

syslog-status
Status of the syslog daemon.

syslog-tail
Shows the last lines.

<- section ("config",section_bo_config);>
}