We've launched our new site at www.openlighting.org. This wiki will remain and be updated with more technical information.
Difference between revisions of "OLA developer info"
From wiki.openlighting.org
(→IOQueue) |
(→MemoryBuffer) |
||
Line 134: | Line 134: | ||
<tt>[http://code.google.com/p/open-lighting/source/browse/include/ola/io/IOQueue.h include/ola/io/MemoryBuffer.h]</tt> | <tt>[http://code.google.com/p/open-lighting/source/browse/include/ola/io/IOQueue.h include/ola/io/MemoryBuffer.h]</tt> | ||
− | A MemoryBuffer is useful if you have a const block of memory, and | + | A MemoryBuffer is useful if you have a const block of memory, and want to extract typed data sequentially using an InputStream. |
<pre> | <pre> |
Revision as of 16:41, 9 March 2013
This attempts to describe the code structure of OLA, in particular the core C++ framework, the olad server and the plugins. Be sure to read Using OLA first to understand the Port, Device, Universe & Plugin terminology.
Contents
- 1 A Brief Tour
- 2 Core Framework Classes
- 2.1 DmxBuffer
- 2.2 Callbacks
- 2.3 CallbackRunner
- 2.4 MultiCallback
- 2.5 Input & Output Buffers
- 2.6 IOQueue
- 2.7 MemoryBuffer
- 2.8 Input & Output Streams
- 2.9 Clock
- 2.10 Credentials
- 2.11 Base
- 2.12 Logging
- 2.13 String Utils
- 2.14 STL Utils
- 2.15 Filesystem Utils
- 2.16 Random Numbers
- 2.17 Network Utils
- 2.18 IP Addresses & Socket Addresses
- 2.19 SelectServer & Sockets
- 2.20 Threads
- 2.21 Mutexes & Condition Variables
- 2.22 Timecode
- 2.23 JSON
- 3 OLAD
- 4 Plugin System
- 5 RPC Mechanism
- 6 HTTP Server
- 7 Ideas for easy configuration
- 8 About device config messages
- 9 The tool app "lla-usbpro"
- 10 Unsupported USB devices
A Brief Tour
Let's quickly cover the layout, you can browse the code online to follow along.
- include - contains the header files (.h) that are installed on the system
- common - code for the ola framework, contains the implementations of everything in include/ola/
- olad - the olad server code
- ola - the ola C++ client code
- plugins - headers and code for all the olad plugins
- javascript - the client side code used for the web UI
- java - the Java OLA client
- python - the Python OLA client
- tools - Various utilities like the RDM sniffer, the RDM Responder Tests, DMX Trigger etc.
include/ola is further broken into modules. There are modules for RDM (include/ola/rdm), HTTP (include/ola), IO, Network etc.
Core Framework Classes
The headers can all be found in include/ola
, the implementations are in common/. While some of these just wrap the native library calls, we abstract them so that it's easier to port OLA to new platforms. Ideally there would be no platform-dependent code outside of common.
DmxBuffer
The DmxBuffer class allows DMX data to be passed around the code, while avoiding unnecessary copying.
DmxBuffer data; // new empty buffer, starts with a size of 0 data.Size(); // size of the buffer, 0 to 512 data.Blackout(); // set all channels to 0 data.Set(data, length); // set the buffer from a uint8_t* data.Get(data, &length); // copy the data buffer into the memory pointed to by data Dmxbuffer data2 = data; // no copy
DmxBuffers are very similar to the boost shared_ptr
Callbacks
Callbacks are similar to function pointers, they allow both functions and methods to be invoked at a later time with data from either / both the time the Callback is constructed and the time the Callback is executed.
All Callbacks have a Run() method, which is how the Callback is executed.
Callbacks come on two varieties, Persistent and SingleUse. SingleUse callbacks delete themselves after Run() is called so you don't have to.
// wrap a function that takes no args and returns a bool SingleUseCallback<bool> *callback1 = NewSingleCallback(&Function0); // some time later bool result = callback1->Run(); // callback1 has deleted itself at this point // create a Callback for Method1 of the Object class and bind TEST_VALUE as the first arg Callback<void> *callback2 = NewCallback(object, &Object::Method1, TEST_VALUE); // this will call object->Method1(TEST_VALUE) callback2->Run(); // this wasn't a SingleUse Callback, so callback is still around and needs to be deleted manually. delete callback2; // create a Callback for a method that takes 1 arg and returns void BaseCallback1<void, unsigned int> *callback3 = NewCallback(object, &Object::Method1); // Call object->Method1(TEST_VALUE) callback3->Run(TEST_VALUE); // callback3 is still around at this stage delete callback3; // create a callback for a method that takes 2 args and returns void BaseCallback2<void, int, int> *callback4 = NewSingleCallback( object, &Object::Method2, TEST_VALUE); // This calls object->Method2(TEST_VALUE, TEST_VALUE2); callback4->Run(TEST_VALUE2); // callback4 is still around delete callback4;
CallbackRunner
Sometimes you need to ensure that a callback is always executed when a method returns.
void Foo(MyCallback *on_complete) { CallbackRunner runner(on_complete); // do work here, which may contain return statements, on_complete will always be executed. }
MultiCallback
Multicallback is a callback that executes another callback after it has been called N times.
/** * Calls DoSomething() for each Port and runs the on_complete callback once each port's callback has run. */ void DoSomethingForAllPorts(const vector<OutputPort> &ports, SomethingCalback *on_complete) { // This will call on_complete once it itself has been Run ports.size() times. BaseCallback0<void> *multi_callback = NewMultiCallback( ports.size(), NewSingleCallback(this, &SomethingComplete, on_complete)); vector<OutputPort*>::iterator iter; for (iter = output_ports.begin(); iter != output_ports.end(); ++iter) { (*iter)->DoSomething(multi_callback); } }
Input & Output Buffers
include/ola/io/InputBuffer.h
include/ola/io/OutputBuffer.h
These define the abstract classes for reading and writing from/to memory buffers.
IOQueue
The IOQueue implements both the InputBuffer and OutputBuffer interfaces.
MemoryBuffer
A MemoryBuffer is useful if you have a const block of memory, and want to extract typed data sequentially using an InputStream.
void ExtractData(const uint8_t *data, unsigned int length) { MemoryBuffer input(data, length); // can now use input with an InputStream. // ... }
Input & Output Streams
include/ola/io/InputStream.h
include/ola/io/OutputStream.h
These allow you to read / write typed data to Input/Output Buffers.
Clock
Contains the TimeInterval, TimeStamp and Clock classes for managing time. It also defines a MockClock class which can be useful for testing.
// get the current time TimeStamp timestamp, timestamp2; Clock::CurrentTime(×tamp); // sleep for a bit usleep(10000); // print the duration we slept for Clock::CurrentTime(×tamp2); TimeInterval interval = timestamp2 - timestamp; cout << interval << endl;
Credentials
include/ola/base/Credentials.h
Functions to get user and group information.
uid_t uid = GetUID(); PasswdEntry passwd; if (!GetPasswdUID(uid, &passwd)) return; cout << "Username is " << passwd.pw_name << endl;
Base
Helper functions that should be called on startup. They do things like install SEGV stack trace handlers, initialize the random number generator etc.
AppInit(argc, argv);
Logging
Contains logging macros which behave like streams:
OLA_FATAL << "foo"; OLA_ERROR << "bar"; OLA_INFO << "baz"; OLA_DEBUG << "bat";
Logging is initialized with a call to InitLogging(level, output). i.e.
// Send INFO and above to STDERR InitLogging(OLA_LOG_INFO, OLA_LOG_STDERR); // or // Send WARNING and above to SYSLOG InitLogging(OLA_LOG_WARN, OLA_LOG_SYSLOG);
Note you can't send different levels to different destinations (so you need one or the other of the above examples). Calls to InitLogging() overwrite the previous logging configuration.
String Utils
While not a class, this defines a number of helper functions for dealing with Strings. If you need to split strings, convert ints to strings and back, escape, trim or capitalize strings use these functions.
STL Utils
Various helper methods for dealing with STL containers like vector and map. Try to use this as much as possible to reduce code (and the change of introducing a bug!).
vector<Foo*> foos; STLDeleteValues(&foos); // delete all objects in foo map<int, string> our_map; vector<int> keys; vector<string> values; STLKeys(our_map, &keys); STLValues(our_map, &values);
Filesystem Utils
A collection of functions for working with files.
vector<string> usb_devices; // return all files that match /dev/ttyUSB* FindMatchingFiles("/dev/", "ttyUSB", &files);
Random Numbers
A simple uniform random number generator. You need to call InitRandom() before using this. Do not use for anything security related.
InitRandom(); int r = Random(1, 10); // returns an int in the range 1 .. 10
Network Utils
include/ola/network/NetworkUtils.h has helper methods for converting between endian formats, and converting IPv4 addresses to strings and visa-versa.
IP Addresses & Socket Addresses
include/ola/network/IPV4Address.h include/ola/network/SocketAddress.h have classes used to represent IP Addresses and Socket Addresses.
SelectServer & Sockets
The SelectServer is the dispatcher at the core of OLA and is defined in include/ola/network/SelectServer.h. It waits for events, and when an action occurs calls the specified method. The SelectServer can also be used to register Timeouts (called every N ms) and Loop functions (shouldn't be used).
Threads
This works as you'd expect.
class MyThread: public Thread { protected: void *Run() { // write code for the new thread here } }; MyThread thread1; thread1.Start(); // blocks until the thread is running // continue, and later thread1.Join();
Mutexes & Condition Variables
This provides Mutexes (non-recursive) and Condition Variables (Monitors).
Mutex mu; int counter; void Foo() { mu.Lock(); counter++; mu.Unlock(); }
While this works, it's not good coding practice. Consider what happens if someone later changes this to:
Mutex mu; int data_to_protect; bool Foo() { mu.Lock(); counter++; if (counter % 10) return true; mu.Unlock(); return false }
Now the Mutex isn't unlocked when counter is a multiple of 10. On the next call the program will deadlock.
To avoid this, we use the MutexLocker class.
void Foo() { MutexLocker locker(&mu); counter++; }
It saves a line of code, and unlocks the Mutex automatically when it goes out of scope.
Timecode
include/ola/timecode/TimeCode.h include/ola/timecode/TimeEnums.h
A simple class for handling Timecode data. See OLA Timecode for more info.
JSON
A set of classes for constructing JSON data structures and serializing them.
JsonObject obj; obj.Add("name", "simon"); obj.Add("Australian", true); obj.Add("age", 21); JsonArray *lucky_numbers = obj.AddArray("lucky_numbers"); lucky_numbers->Append(2); lucky_numbers->Append(5); // returns the above as JSON. const string json_output = JsonWriter::AsString(obj);
OLAD
RPC System
The RPC system is what allows clients to communicate with the OLA Server. It uses Protocol Buffers as the data interchange format. You can see the message definitions in the Ola.proto file.
Clients communicate with OLAD over a TCP socket connected to localhost::9010. The RPC port olad listens on can be changed with the --rpc-port command line option but this will require the clients to be updated as well. Because the communication is over localhost we don't need to worry about dropped messages causing the socket buffers to fill up. You can still exceed the socket buffers if the client sends faster than the server can receive, but this usually indicates a poorly written client.
Client Side
The code required on the client side is the similar irrespective of langage. All that's required to write an OLA client is an implementation of Protocol buffers. See the Protocol Buffers, Other Languages page.
Client code needs to:
- Open a TCP connection to localhost:9010
- Provide an API, which constructs protocol buffer objects, serializes them and writes them to the TCP socket.
- Read data from the TCP socket, de-serialize it into protocol buffer objects and deal with the response data.
The only tricky bit is matching the responses to the requests. This task is usually handled by a StreamRpcChannel class, see the Python or C++ code.
Server Side
On the server side, OLAD creates a listening socket and then calls InternalNewConnection each time a client connects. This sets up a OlaClientService object, which is what gets called when a new message is received. The OlaClientService objects call into OlaServerServiceImpl.cpp which is where the message handling takes place.
Plugin System
We'll use plugin to refer to the entire module (Plugin, Devices & Ports), and Plugin to refer to the class that inherits from Plugin.
Plugins create and register Devices, which each consist of 0 (obviously not useful) or more Ports. A Plugin generally does a bit of work when it starts to detect devices, then leaves all work to the individual Devices and Ports.
Each plugin implements the classes in blue. Of course, you can choose not to inherit from the BasicPort / Device / Plugin classes and do everything yourself.
PluginAdaptor
The PluginAdaptor is the interface between plugin code and the core OLA objects. Each Plugin object has a pointer to a plugin adaptor in the instance variable m_plugin_adaptor.
Plugins
The AbstractPlugin interface is defined in include/olad/Plugin.h. The Plugin class implements most of this interface, and leaves Id(), PluginPrefix(), StartHook(), StopHook(), SetDefaultPreferences() and Description() to be implemented by the child classes.
The startup sequence for a Plugin object is:
- From within DynamicPluginLoader::LoadPlugins an instance of the plugin is created
- If the ShouldStart() method returns False, nothing else happens, otherwise the Start() method is called.
- The Start() method calls LoadPreferences() which in turn calls SetDefaultPreferences(), this last method gives the Plugin the opportunity to setup the Preferences object.
- if SetDefaultPreferences() doesn't fail, StartHook() is called where new Devices are created. m_plugin_adaptor->RegisterDevice() should be called to add new Devices.
During the shutdown sequence:
- Stop() is called, which in turn calls StopHook()
- StopHook should call m_plugin_adaptor->UnregisterDevice() for any devices registered during the start phase.
- delete is then called on the Plugin object
At any time a the following methods can be called:
- Id()
- Name()
- Description()
Devices
The interface to Devices is defined in include/olad/Device.h as AbstractDevice, again the Device class implements most of this interface, leaving the derived classes to fill in a couple of methods:
- DeviceId() - returns a unique persistent string identifying this device
- StartHook() - this should create the port objects for a device.
Ports
Ports are the objects that actually read/write DmxBuffers. Defined in include/olad/Port.h there is the base interface Port, and then two child interfaces, InputPort and OutputPort. The BasicInputPort and BasicOutputPort provide partial implementations for these two interfaces.
At a minimum , an OutputPort needs to provide the following methods:
- WriteDMX(const DmxBuffer &buffer, uint8_t priority)
- Description()
And an InputPort needs to provide:
- ReadDMX()
- Description()
A call to ReadDMX() is triggered by calling DmxChanged() on a InputPort object. This causes the universe the port is bound to to fetch the new DMX data. Both ReadDMX() and WriteDMX() must be non-blocking, blocking here will delay the main processing loop. To satisfy this, most ports use this sequence of events:
- register a Socket for reading with the SelectServer
// some time later
- receive notification that there is new data on the socket
- read the data and copy it to a buffer
- call DmxChanged() to notify the bound Universe we have new data
- the Universe then calls ReadDMX()
Often more than one port will use the same file descriptor. This means the device is responsible for reading the data and dispatching to the right port.
Here's an example of how dmx data is received from the UsbPro Device.
The UsbProDevice will have been registered using plugin_adaptor->RegisterSocket(). When input becomes available the following sequence happens:
device->action() // signals the device that new data is available widget->recv() // tells the widget to read more data widget->do_recv() // reads the data from the fd widget->handle_cos() // handles the change-of-state message from the widget device->new_dmx() // signal the device that new dmx data has arrived port->DmxChanged() // signal the universe that new dmx data has arrived // if this port is bound to a universe, the universe will then call port->ReadDMX() device->get_dmx() widget->get_dmx()
Of course, the plugin authors are free to implement this however they like.
Config Messages
Config messages are handled a little differently for two reasons:
* The configure() method in a plugin has to return a response immediately. We don't want to block because we'll delay all lla processing. The new RPC subsystem removes this limitation. * Sending a PARAMETER_REQUEST to the widget doesn't generate a response immediately (in fact it may not generate one at all).
To work around this, we send a parameter_request when we start the device, and then anytime we set parameters. In the meantime we store the parameters in the widget object and return those. The sequence looks like:
device->configure() device->config_get_params() widget->get_parms()
- What is the interface between the LLA core and LLA plugins?
See above and the files plugin.h, device.h and port.h. The create() call will be passed a PluginAdaptor object which can then be used to register/unregister file descriptors, loop functions, timeouts and devices.
- What is the interface between the LLA core and other apps/clients to LLA like QLC?
All clients should use the LlaClient library. This needs better documentation.
- How is functionality split between the usbpro plugin and the example program?
The example program constructs configuration request messages and sends them (using LlaClient) to the Lla Core. The core routes this message to the plugin, which then returns a response message. This response is passed back to the client.
RPC Mechanism
RPCs are at the heart of OLA. The RPC system is built on protocol buffers. olad/OlaServerServiceImpl.h provides the implementation for the RPCs on the server side.
HTTP Server
The HTTP server is provided by microhttpd. The ola/http/HTTPServer.h library provides a C++ class wrapper around the microhttpd code, and then olad/OlaHttpServer.h implements the OLA specific behavior.
Ideas for easy configuration
For some users, it will be useful to have a "auto-connect" feature. When a attached device is discovered (either when LLA i started or when a new device is attached), the user could be asked if the available ports (input as well as output) should be patched to the lowest available universes.
- Enable auto-connect ( OFF|connect whatever comes first|connect by stored patch layout)
- Save a given combination of devices (just by type or with unique ID's from serial numbers, USB device ID's etc)
Which devices cannot be autodetected?
About device config messages
We need a way to tune settings on a port/device that the LLA Core doesn't know about. To enable this, the LlaClient provides a method dev_config(unsigned int dev, LlaDevConfigMsg *msg)
The LlaDevConfigMsg is an interface which declares one method: pack(uint8_t buffer, unsigned int length). On the device side, we declare a method configure(uint8_t *request, int length)
So to use this:
On the client
MyObserver::dev_config(unsigned int dev, uin8_t *res, unsigned int length) { MyLlaDevConfigMsg msg = parse_message(data, length); // do something with the result } int main() { // all the setup code MyObserver observer; // the observer gets the dev_config() callback lla_client->set_observer(&observer); MyLlaDevConfigMsg msg; // set some fields msg.foo = 1 lla_client->dev_config(device_id, &msg); //calls pack() on the message }
In the device:
MyDevice::configure(data, length) { MyLlaDevConfigMsg msg = parse_message(data, length); // do something with the message MyLlaDevConfigMsg *response = new MyLlaDevConfigMsg(); // response is deleted by the lla core return response; }
The tool app "lla-usbpro"
The purpose is to set and get the settings that reside in the USB Pro box.
The communication with USB Pro's seems to go via the LLA core, and lla-usbpro registers as a LLA client, and uses some event handlers.
As defined in the device spec. (PDF from Enttec):
label=3 response
- 1. data byte= Firmware version LSB. Valid range is 0 to 255.
- 2. data byte=Firmware version MSB. Valid range is 0 to 255.
- 3. data byte=DMX output break time in 10.67 microsecond units. range=[9-127] (96.03 - 1355.09 micro seconds)
- 4. data byte=DMX output Mark After Break time in 10.67 microsecond units. range=[1-127] (10.67 - 1355.09 micro seconds)
- 5. data byte=DMX output rate in packets per second. range=[1-40]
- x. data byte= some user configuration of the requested size
The serial number is is decoded (from 4 bit Binary Coded Decimal) in lla-usbpro, not the plugin.
Unsupported USB devices
- Peperoni Rodin1 (but this is supported directly by QLC)
- Peperoni USBDMX21
- USB DMX from usbdmx.com
- Sandsys UMX2
- Sandsys UMX4
Peperoni and usbdmx are probably easy to implement. The specs and source code examples are available.