This site is deprecated; docs have moved to!

Difference between revisions of "Historical:Development/RemoteIncludes"

From the makers of InspIRCd.
Jump to: navigation, search
(Remove dead link.)
m (SaberUK moved page Development/RemoteIncludes to Historical:Development/RemoteIncludes: Move to the Historical namespace using SaberWikiBot.)
(No difference)

Latest revision as of 17:03, 21 May 2015

Development Development Material - Information posted here is for developer reference only. This material is subject to possible change and will be technical in nature.

Remote Includes


The goal is to be able to securely fetch configuration files on demand from a central server. This makes management and central control of servers on a network MUCH easier.


The running theory is to use a combination of m_httpd (on a hub / the server which will serve configuration files) and m_http_client (on all other servers) - which has the benefit of not requiring a securely configured 3rd party server. These can also be extended to support SSL and some form of authenticity check (SSL certificate fingerprints?). Each server would then send a request with m_http_client to get it's files, before loading the configuration (during rehash, etc).


The major issue with this idea is a 'chicken-egg' situation on startup; the configuration needs to be fetched, but can't be fetched until some things are configured and initialized. Notably, the socket engine must be fully operational (outgoing connect) and configured (IP binding), SSL must be loaded and configured (if used), m_http_client must be loaded, and whatever module is required for remote includes must be loaded and configured. There are LOTS of values required for the IRCd to run that aren't reasonable to require in the local configuration.

Solution #1: Revise configuration loading

One solution would be to somehow revise configuration so that the ircd can be partially started (enough for the conditions listed above), without requiring the majority of it's configuration. This would, however, be fairly complicated to do, and rather annoying.

Solution #2: ...?

How about using a "mini-ircd" (for lack of a better term), or perhaps just using wget if authentication is not needed, in order to download remote includes when the ircd starts? After everything is up and running, the hub server could instruct the rest to download whatever parts have changed and rehash automatically. The remote includes could optionally be stored locally to prevent having to start the mini-ircd, as well as to provide a last known configuration should the hub server not be available. --Dz 23:09, 3 September 2007 (UTC)

Solution #3 (based off 1)

This seems to be most popular currently:

[21:14] <Brain> but it means module rejigging again
[21:15] <Brain> we make all modules so that they dont read their config tags in their constructor
[21:15] <Brain> but when instructed in an OnReadConfig event :p
[21:15] <Brain> then, we just trigger the OnReadConfig event well after all the modules are loaded correctly :D
[21:15] <Brain> (and the config is re-read) :)
[21:15] <Brain> basically two pass
[21:16] <Brain> sound good?
[21:16] <Brain> PASS ONE: read all config, load all modules, verify its all ok. DONT call OnReadConfig in each module
[21:16] <Brain> PASS TWO: read all config again, call OnReadConfig in each module
[21:16] <Brain> note: during pass one, when each module is loaded it will Attach() all its hooks
[21:17] <Brain> which means any remote includes scheme implementors will hook the OnRemoteInclude or whatever, and pick up *their own* config data for a later call to OnReadConfig
[21:17] <Brain> how smart is that :D
[21:17] <Brain> this does mean that your <modules> tag which loads the remote include module must be in a local config
[21:18] <Brain> as must enough to boot the ircd, e.g. its name, etc
[21:18] <w00t> that sounds very good. :)
[21:18] <w00t> well t hat makes sense
[21:18] <w00t> but then the rest of the modules can be elsewhere
[21:18] <Brain> but stuff like opers, which arent 'required' elements, will be fetched during pass 2
[21:18] <Brain> yup
[21:18] <Brain> that'll be read and applied during pass 2
[21:19] <w00t> though
[21:20] <w00t> how do modules unload then if their config/load fails
[21:22] <Brain> easy
[21:22] <Brain> return value from OnReadConfig


The config reading is now a two-pass read. The first pass reads the bare bones configuration enough for modules to be loaded. Modules do not load configuration at this point. The remote include module implements an OnDownloadFile which can start a file download. The core then waits on the socket engine until Config->Downloading() returns false, at which point it initiates pass two of the config parsing, finalizing settings and then calling OnReadConfig in each module, which causes it to read its config.

Any files in the file:// scheme (or without scheme) are implemented by the core. This is still considered 'file downloading' however it is an instant, blocking operation.

Multiple modules may implement OnDownloadFile (e.g. if someone wants to write a remote includes system that uses exec() to fork rsync, or other such craq)

The remote include module fetches each file asyncronously in a non-blocking manner, flagging each file as complete by name by calling the Config->Complete(std::string name, bool error) method.

Before initiating pass two, if any of the files have errors (there is planned support for a more detailed error message) then the reading of the config is aborted.

Remaining to be done

  1. Move all modules config reading to OnReadConfig. At first glance there are ~80 modules that require this
  2. Make m_http_client implement a proper interface api, so that we can enumerate it and any other future ones (m_ftp_client, anyone?)
  3. Fix the (relatively minor) issue of an include within an include not working properly yet.
  4. Possibly have ConfigReader check the current configuration pass number in its constructor -- if a module attempts to load the configuration from its constructor (they shouldnt!) throw a CoreException("Badly behaved module, update to use OnReadConfig") -- we should do this before point 1, possibly