RFC: Sensor driver interface #2093
Comments
|
The key to tackling fixedp vs floatp problem would be adopting design which doesn't have methods like .temperature(), acceleration(), etc. Instead, all sensor classes should inherit from the following base classs: This makes it clear how it all works: default and recommended way for sensor drivers is to work and return fixed-point values. However, end users wishing to work with floatp, will be able to thanks to the base class. And drivers who really can't work with floatp, can override get_float() directly. This also shows that representation of fixedp should be just normal int, as anything else will require allocation. It's unclear if we will be able to use the same fixed point position for all physical quantities, but it's good target, and then apparently we should be down to 1/1000th. This of course requires adopting a rule that base value is not raw Si unit, but may be a scaled one. Important to ensure user-frindliness though is adopting a rule that then only standard decimal Si prefixes can be used as a base unit, e.g. pressure is measured in kPa, but not 10,000s of Pa. |
|
Other important point showed in the base class above is that physical reading of a sensor and getting a result should be separated. Besides other reasons, this stems from no-alloc rule. E.g., an accelerometer will measure values for X, Y, Z, but returning them all at once would require a tuple, requiring an allocation. So instead, 3 separate call for get_fixedp(ACCEL_X), get_fixedp(ACCEL_Y) should be done. This will allow (and oftentimes require) caching of values. As further optimization, for compound sensors, .measure() method may accept a limit list to skip measuring (or calculating) particular physq it is otherwise capable of, with default being "measure everything". |
|
Perhaps the last point of this initial picture is how to specify "what" param in the code above. We'll definitely need to make an inventory of measurable quantities and base units for them, and as usual, I'd start with oldskul idea of using symbolic numeric constants. Note that if we adopt idea of passing bitmask of them to .measure(), it means that we can support only 31 quantity, which is perhaps an arbitrary restriction, so some of the interfaces above need adjustments. |
|
I tried to rewrite my BMP180 library to meet this model. See here. My thoughts:
Upon first reading I found this to be kinda weird, after implementing it, I find it pretty elegant. Makes for a nice and portable base. Sensor reading is unified and simple. Sensor configuration just has to be in a good default state, changing it will be very device specific. |
|
Are there not ways to return multiple values without allocation? Pass an array or list as a function argument or have the class constructor instantiate an array or list and return a reference to it. |
|
@turbinenreiter: Thanks for looking into it and even trying. My comments are below.
This is inherently complex problem, and needs to be considered separately. I'd say the initial proposal above covers just blocking mode. "Non-blocking" mode support would require its own design phase, starting with usage stories, then with requirements. You show an interesting implementation approach, but it's already clear that it essentially duplicates a cooperative scheduling loop. Why does it do so? Should it be converted to Python standard cooperative scheduling loop (asyncio) or stay that way? Only user stories put into the design of the programming interface may answer.
How is that less pretty if it explicitly clarifies the namespace of a symbol? I'm afraid, there's no modern language without namespaces. Otherwise, how to use them depends on API designer, as usual. I particularly have in mind a global registry of measurable quantities defined on module level (e.g. of the module which contains SensorBase definition), so there won't be the need to have sensor.ACCEL_X.
Is calculating altitude specific to pressure data of a particular barometer sensor? Physics says us that not, and then barometer drivers should not expose ALTITUDE datapoints. Instead, it can be a separate adapter class (likely, a function) which will work with any barometer sensor. |
Yes, I'd say these are out of scope of the current proposal (I've added new p.1 to requirements clarifying the implied user story behind this RFC). Of course, feel free to think about that too and share considerations. Mine would be that it's similar problem to other API domains. We've come to a good general solution in network interface API - .config(param=value) method. I'd suggest considering the same here too. |
Of course there're ways. Feel free to present user stories behind a need to return multiple values at once, then how it all fits together, then how it fairs with "There should be balance between efficiency tricks and pythonic ease of use." requirement. |
|
I've updated according to your feedback.
I also added a 'non-official' disclaimer to my SensorBase file, to make sure people know that this is just an ad-hoc implementation to experiment with ideas coming up in the discussion. |
I agree with @turbinenreiter : in the case of a sensor which returns a vector, it seems more logical to have a single method returning a 3-list or array than have methods for x, y and z. |
|
Another aspect is documentation. I found it useful to have a link to an application note or datasheet. Also typical usage examples help a lot ( e.g. a REPL session). In order to minimize ROM/RAM footprint of the driver, such documentation could belong to a wiki entry (or possibly a subchapter in the official doc?) so only the link remains in the driver code. |
|
+1 on leaving vector data together. Maybe I'm missing something here but what is the advantage over a simple "read_temperature()"? E.g. measuring a temperature value would become something like sensor.getfixedp(sensor.TEMPERATURE). Cant we just agree on a set of read_xxx methods? I found it useful to instantize a sensor on the repl and by "Tab autocompletition" find the method that the sensor exposes. I would miss this feature if it only exposes a generic measure function. Fixed point vs floating point: Is it worth the effort? Don't we sacrifice python simplicity (to which floating point belongs) to optimisation? Isnt 19.4 deg more intuitive than 194 decidegrees? |
... so you propose each driver to handle that on its side? That's a lot of repetitive code. Also, to do optimal scheduling choices, some global knowledge is required. For example, if the main loop knows there's no more work to do for next 5ms, it can put CPU to sleep, instead of looping for nothing. If you implement such stuff, that you essentially duplicated (u)asyncio main loop. uasyncio can also be optimized, e.g. implemented in C (which can likely happen given the stretch goal of esp8266 support), and any users will benefit. All in all, as I mentioned, there's no easy solution right away, and I guess it would be beneficial for @dpgeorge to see how people approach it now, I hope he'll look at the previous version you had.
That's so unfortunate of them. Because this RFC treats all sensors the same, with differences only emphasizing the similarity. In particular, this RFC treats a "compound" sensor not worse than a set of sensors each handling just one quantity. Contrary, trying to add adhoc handling of compound sensors immediately affects simplicity of the API, and poses bunch of questions, like: how does user no which compound values may be returned as array? In which order? What if user wants subset and/or different order? I extend my invitation to provide complete picture how it all fits end-to-end.
I'd say not exactly. That's why I asked about a user story. I guess it's not just "I want to get measurement results ASAP as they are ready", but "I'm building a quadrocopter and the lowest possible latency of readings is critical for its ability to fly well at all". But then chances are that a user with such user story won't be interested with "drivers" at all, but will start with a particular accel/gyro model and adhoc code to make sure it flies at all, then with another accel/gyro and adhoc code, and only then some kind of special-purpose driver model can proposed, driven by adhoc usage. Or if it can be said right away that those kinds of sensors still have millisecond-level delay, then there shouldn't be concerns at all - 3 calls to a simple function and array assignment still will be on the level of microsecond.
To clarify, this is nothing but a proposal which records already known concerns (like fixed vs floating point), sets requirements based on them, and provides initial ideas how to handle them. It's mostly to record the thoughts, ask for feedback, and be initial input for @dpgeorge who will handle it a later time (per our default hardware vs networking split, though we may switch tasks too of course). |
|
@pfalcon: sorry but your last post came in before i did finish this one. So please bear with me in case i do write obsolete stuff! Before starting to over-engineer the simple acquisition interface we should look at the real world use-cases. Reading a sensor data is only one part of the show. How about interface guidelines for additional sensor functionalities? For example there can be interrupts triggered by an alarm feature of a sensor (e.g. passing temperature thresholds or illumination levels). Reading multiple sensor data values at once might be as reasonable as returning and reading single features of a sensor only. Often timing penalties for reading all features can be avoided by reading one feature only. For simple sensor reading there should also be suggestions regarding the different settings when acquiring sensor readings: e.g. accuracy can and should be given in additional keyword arguments. Regarding the floating point discussion i would say that one should simply implement the driver with no memory allocation in mind. So the only way is to avoid floats or use a floating point emulation/workaround. This is imho a global task to avoid repetitive and possibly bad code. All sensors should be able to use this kind of float emulation/wrapper. Just some of my cents on sensor driver interfaces... g'night all! |
|
If we'd continue this discussion and/or come to a decision, i would be glad to implement some of my drivers accordingly! |
|
@kfricke : I believe previous messages clarify intent of this ticket, it's scope and how extension to it can be handled. As it is now, it doesn't represent any established API, and in my list, wait for response from @dpgeorge (though alternative - consistent and more or less elaborated - proposals are welcome from other interested parties). If you like the current proposal, you're welcome to explore it by trying to implement drivers along it lines, but it would be just a research work, again, it's not suitable for anything "production". (For example, if @dpgeorge or someone else comes with a better proposal which covers all the current requirements, and more). Note that if @dpgeorge doesn't respond, then he's busy with other things and it make take him quite awhile to get to this topic. And that's why I created this topic - to record my thoughts, which a based on months of previous discussions in tickets and on forum - to not forget these thoughts, as I'm working on (a lot of) other things in the meantime. |
|
Thank you and i am aware of your workload and the different higher priority tasks. But thanks for the time to give this feedback! |
|
Sorry for the late answer, but I decided that my initial comment here was not very constructive and useful, and decided to wait until I'm back home and can actually contribute something useful. I just have two comments. First, I think that if we will start by a completely theoretical discussion like here so far, then we will come up with a design that is very complex, very flexible, hard to understand and that doesn't actually cover the common use cases. I think the best approach would be to actually pick a couple of sensors, write the drivers for them, try to use the drivers in practice and then modify them until we are happy with the API. Second, I recently wrote a driver for the VL6180 sensor for Micropython, with a very ad-hoc API, (https://bitbucket.org/thesheep/micropython-vl6180/src/tip/vl6180.py) and I noticed that the chip actually has much more inputs than outputs -- it only measures distance and ambient light, but it has a lot of knobs to tune it for the optimal measurements. I think that a good sensor API should include the methods for configuring the sensors. I understand that writing the drivers for a lot of sensors up front, and then modifying them all to make them follow a common API is a lot of work. Perhaps instead we could look at some sensors commonly used in projects, and look at the drivers available for them, for instance for Arduino. I think that the Adafruit's github repository (https://github.com/adafruit/), as well as Sparkfun's (https://github.com/sparkfun/) are excellent sources of such drivers, and could be used to learn a lot about the practice of using sensors. |
|
I wanted to point out that in Python it's not necessary to have separate functions for two different return types, like int and float -- the same function or method may return int or float depending on a flag passed to it. While it is normally considered a suspicious practice to have a function return different types, in this case it is fine, because the caller has to explicitly state the flag, and so they know what type they are going to get back, so there is no surprise here. |
|
In the mean time I wrote a number of sensor drivers for MicroPython, perhaps the code will be useful for considering actual use cases. I tried to follow existing APIs whenever possible.
|
|
In particular, I would like to point your attention to how the tcs34725 driver lets you do both blocking and non-blocking reads with this code: https://github.com/adafruit/micropython-adafruit-tcs34725/blob/master/tcs34725.py#L100-L114 By the default, if you just do |
|
Would it make sense to add |
|
Sorry for not joining this discussion earlier. I would say that the number 1 requirement is for the sensors to return SI units. This is so you can easily combine readings from multiple sensors (eg 2 different temp sensors, or time and distance sensor to produce speed) without having to use any conversion factors which could introduce many bugs. The number 2 requirement would be that you can swap out one sensor for another if it measures the same thing. The only thing you might need to change is the constructor/initialisation code. The measurement/reading code should stay the same, and the result returned should have the same SI unit. Some sensors have a very wide dynamic range, or at least the possible values to measure for a given physical quantity can have a large range in realistic scenarios. For example:
Floats are perfect for this kind of situation, and maybe we just need to live with the fact that sensors may allocate memory in computing and returning their value. An alternative to FP would be to make a kind of integer floating point return value, like a pair of integers |
|
@dpgeorge You might want to access sensor data in an interrupt callback. Notably a timer callback: for some types of filtering and analysis you need accurately timed samples. What is your view on the earlier discussion about sensors which return vectors? Keeping vector elements together can be important. You want to be sure when sampling a sensor such as a gyro that the elements correspond to a single time instant. If you have to request each orthogonal axis separately, this correspondence is lost. |
A vector is a physical quantity (ie it has meaning) and if a sensor can measure a vector quantity then it goes without saying that one should be able to get all axis corresponding to a single measurement. Then you can do things like calculate the magnitude, or dot product, etc, using a consistent vector. But I think @pfalcon's proposal allows this, since the measurement phase is done by one call, then you can get the individual axis, one-by-one, corresponding to the single measurement. |
|
I wonder if sequence unpacking + generator function could be used to return multiple values without memory allocation? Something like: I suppose calling a generator function already allocates memory? |
It might work... but a simpler way would be to use uctypes, something like: import uctypes
class Sensor:
def __init__(self, ...):
self.buf = bytearray(6)
self.vector = uctypes.struct(uctypes.addressof(self.buf), {'x':(uctypes.INT16 | 0), 'y':(uctypes.INT16 | 2), 'z':(uctypes.INT16 | 4)})
def get_vector(self)
# read raw sensor data into buffer
self.i2c.readfrom_mem_into(self.i2c_addr, self.mem_addr, self.buf)
# return already-created structure which points to buf
return self.vector
s = Sensor(...)
vec = s.get_vector()
print(vec.x, vec.y, vec.z) # unpacking of struct is done here, and doesn't allocate any RAMNote that the underlying storage format of the buffer can match the raw data from the sensor, and uctypes will covert it for you. |
A good start for such a list is Adafruit's unified sensor interface, with the list of supported sensors here: https://github.com/adafruit/Adafruit_Sensor/blob/master/Adafruit_Sensor.h They base their sensor interface, and list of sensors, on the Android one here: https://github.com/android/platform_hardware_libhardware/blob/master/include/hardware/sensors.h In Adafruit's list they have 17 sensor types, of which it looks like 8 of those are represented by a 3-vector (eg gyroscope is a vector of, and colour sensor has r, g and b). So if we needed to provide an "enum" for each axis that would already be 33 different values, out of range of 31-bits. Another thing to consider is if a device has multiple of the same sensor types, eg a device with multiple temperature sensors, or multiple force/pressure sensors spread over an area. In order to retrieve all the measurements it seems inefficient to do multiple (eg 10s of) calls to the driver (and how would one enumerate all the individual sensors?). It might be more natural to return a vector with all the values, and this can be done without requiring heap allocation by using either a pre-allocated array or uctypes. |
Having blocking only behaviour will be quite limiting in some common cases. Eg 1-wire temp sensor DS18B20 requires 750ms to make the measurement. So if you have more than one on the wire then you are going to be waiting a long time to read all of them. Instead it would be convenient to just make measure() non-blocking, ie just start the measurement, and then when the value is requested the driver would wait any additional time for the measurement to complete. That way you can loop through all sensors, call measure(), then loop through them all a second time to read the values. |
|
Regarding implementations of the sensor drivers for the ESP8266 stretch goals: pretty much all the initial work has been done on this:
So it just remains to decide on a common interface and a place to commit them. Note that most of the above sensors have some important internal settings that should be exposed to the user (eg acquisition rate, gain, sensitivity, integration time, ADC precision). |
|
Based on discussion above, I would consider the following properties to be important for a general sensor driver to support:
And here is my initial proposal for the API. First, I propose that the non-FP values returned by the sensors are fixed point, with a decimal and exponent (d, e), and encoded in an integer using 8-bits for the base-10 exponent, and remaining bits for the decimal, as follows: def fixed(d, e):
return (d << 8) | (e + 128)
def fixed_to_float(fx):
return (fx >> 8) * 10**((fx & 0xff) - 128)Eg for 20.5 degrees one could write: Sensor methods follow. Configuration is necessary for some sensors (eg light sensor) and we can follow the way network interfaces are configured: sensor.config(setting=value[, setting2=value2, ...]) # configure the sensor, eg rate, gain
sensor.config('setting') # retrieve the value of a settingSettings and values are sensor-specific, eg Measurement often takes some non-trivial amount of time and it makes sense to split it into a start and end call: sensor.measure() # start taking a measurement/reading of all sensors; non-blocking
sensor.end_measure() # finalise reading of sensors; may blockThe idea is that The accessor methods return the readings of the sensor, taken at the most recent call to sensor.get_temperature_fixed() # scalar fixed value in degrees C
sensor.get_temperature() # scalar float value in degrees C
sensor.get_acceleration_fixed() # vector of fixeds in m/s^2
sensor.get_acceleration() # vector of floats in m/s^2For vector return values not to allocate on the heap, the sensor is expected to have an internal vector/array/list to which it assigns the vector values and then returns it. This vector/array/list will be reused for the next call so it's up to the caller to copy out the data if it's need after the next measurement is taken. All the floating-point methods can go in a base class so they don't need to be implemented again and again, eg: class SensorBase:
def get_temperature(self):
return fixed_to_float(self.get_temperature_fixed())
...If a sensor wants to implement multiple sensors together then it can do so. UsageExample usage: hdc = HDC1008(i2c)
while 1:
hdc.measure()
print(hdc.get_temperature(), hdc.get_humidity())
time.sleep(1)Variations to the aboveOne could change the accessor methods to name the return type in the method name (scalar, vector, etc) and then use an enum (or string) to specify the physical quantity. Eg: This scheme would fix the number of accessor methods to about 8 (scalar, scalar_fixed, scalar_array, scalar_array_fixed, vector, vector_fixed, vector_array, vector_array_fixed) at the expense of introducing a bunch of enumerations. Provision should be made for user defined enumerations so users can add arbitrary sensor quantities. |
|
I will try to apply this to one of my sensor drivers, https://github.com/adafruit/micropython-adafruit-ads1015/blob/master/ads1x15.py and see how this works API-wise: Because you have to specify on which pins you are going to be doing the measurement, and whether it's absolute or a difference between two pins, that needs to be specified in the configuration of the sensor and changed before every measurement. Since the gain setting might get changed between calling There is an awful lot of boilerplate code in the Some questions:
|
IMHO, this is overcomplication. A sensor which measures current in fractions of microampere is rather different from a sensor which measures kiloamperes. In no way you can substitute one for another. And both different from a "normal" sensor which measures values of hundreds of milliamperes. Substitution is also questionable. So, these 3 are different sensors types, even though they measure the same physical quantity. The same situation would be with sensors for other physical quantities. So, could define 2 or 3 ranges a sensor would belong to, with different units for each (e.g. a picoampere, 1/10 milliampere, ampere for example above). |
Only that? What if I want callback when measurement is done? What if I want a task to be woken up or scheduled when measurement is done? What if ...? Sorry, but this is the same fiasco-style design process as with the hardware API. Please don't try to do everything at once, it won't work, and only mess will ensue. There're enough questions to resolve even with juts nice simple blocking API. |
For what reason there's a desire to fill in users' memory with repetitive method names? What's the specific answer to this proposal: #2093 (comment) ? |
It's definitely a good idea to look at existing art, here're more links:
|
How's that less complicated than SI-units + base-10 exponent? And why stop at pico and milli? What about nano and kilo? Mega? There is 16 SI-prefixes and even if you implement all of them, y*10**x still gives you better resolution, as the prefixes are always 3 exponents apart.
That's why non-blocking for now only is about basic non-blocking. Why think about callbacks and scheduling, only mess with ensue. Truth is, a simple non-blocking functionality is needed right now because having only blocking reads makes the drivers useless. If you write a standardized BMP180 driver that only works with blocking reads, people will use the existing non-standard one that has non-blocking reads. What do we win by that?
The reason is PEP 20: Readability counts |
|
There has seemingly been no motion on this for over 5 years. Is a generic sensor API even considered to be in scope for core MicroPython? It seems to be something that could very well be done outside the core project - at the very least, all the sensor driver implementations should probably be third-party - since it is practically an infinite amount of them... |
Uh oh!
There was an error while loading. Please reload this page.
As one of the latest KS stretch goals, we need to implement suite of drivers for number of sensors, and it would be nice to come up with a general sensor driver model to follow.
Requirements:
The text was updated successfully, but these errors were encountered: