Wow. This is great. Thanks for taking the initiative. Could you explain a bit more about what you mean by "combine multiple other methods (e.g. have a year intensity method multiplied with a daily sine)" and "Scripted method that allows user scripts to define the calculation/plot", perhaps with examples, so I can get a better idea of what your changes allow? Also, does this break any functionality with users' current method database entries and if so, do you think upgrading their databases to be compatible with your new system would pose an issue?

Code changes

I managed to finish my changes so that also the pid controller works again. Based on reference search I made sure to refactor and test all places, where the old method implementation was used. You can look into it using the link above. If you want, I send a PR. Optionally you can also create a branch that I can pull into, so you can do some testing first. I will comment the changes in the pull request or we have a quick screenshare call so that I can explain them to you directly.

The changes I made are pure refactoring so far. No functionality or database changes are expected. The hope is, that no-one would even notice they are in.

However, they make it much simpler to add new methods. Let me give an example:

class EntityMethod(AbstractDailyFormulaMethod):
    """
    The entity method returns one, regardless of the input. It is actually f(x) = 1
    """

    def calculate_setpoint(self, now, method_start_time=None):
        # Calculate sine y-axis value from the x-axis (seconds of the day)
        return 1., False

Here you are, you created new method. It will also be rendered in the frontend automatically, as I migrated the frontend calculation code into the class structure:

class AbstractDailyFormulaMethod(AbstractMethod):
    """
    Abstract base for mathematical function based methods. It offers shared functionality to generate the frontend
    plot by iterating through the x axis and calling the calculate_setpoint function to get the corresponding y values.
    """

    def get_plot(self, max_points_x=700):
        result = []

        seconds_in_day = 60 * 60 * 24
        today = datetime.datetime(1900, 1, 1)
        for n in range(max_points_x):
            percent = n / float(max_points_x)
            now = today + datetime.timedelta(seconds=percent * seconds_in_day)
            y, ended = self.calculate_setpoint(now)
            if not ended:
                result.append([percent * seconds_in_day * 1000, y])

        return result

So each method defines itself, how it is plotted.

I didn't really change code or how it works but mainly shifted it around massively. If you look into the code, you'll find your code.

One would still have to add it here and there in the frontend though, but this should be possible within a reasonable amount of time.

One other improvement is, how the 'Duration' method works. This was a bit woven, as the database updates for ending and restart were spread across multiple classes and really implemented only for 'duration' type. I cleaned this up by moving all DB related stuff into the controller and added a should_restart() function to the Method class. When a method finished (e.g. last entry in duration table was done) and is not ended (e.g. the method_end_date entry is not reached), the controllers ask should_restart() if a restart is desired.

This is how the "DurationMethod" class responds. You'll identify your code:

def should_restart(self, now, method_start_time=None):
    """
    If a method has signalled to be finished, this method is asked if the controller should restart
    the method processing from the beginning.
    :param now: point in time to calculate the value for
    :param method_start_time: when this method started. Must accept datetime and strings
    :return: True if the method wants to be restarted, otherwise False
    """
    for each_method in self.method_data_all:
        # If duration_sec is 0, method has instruction to restart
        if each_method.duration_sec == 0:
            return True

    return False

PS: A fresh pair of eyes and testing would be appreciated.

Scripted Methods

Your idea of methods (as I understand it) is, that a controller gets a value for a given point in time. Like f(t).

You've got some function in there, like f_dailysine(t) = sin(t) or f_duration(t) = 0<t<10 -> t; 10<t<20 -> 10+t*0.5

The idea is, if we now add a Method class like this:

class ScriptedMethod(AbstractDailyFormulaMethod):
    """
    The scripted method returns values based on user code
    """

    def calculate_setpoint(self, now, method_start_time=None):
        # Calculate sine y-axis value from the x-axis (seconds of the day)
        return eval(usercode)

And have one of the code editors you have all over the frontend available, a user could create an own scripted method by adding python code.

Or read a constant from a system variable or simply return a constant, which gets interesting combined with the next idea below.

Cascaded Methods

Even with the methods in place today, I would like to be able to combine them. Assume you're growing lettuce from seed to harvest. That's usually a more or less plannable thing. You might want to start with less nutrients or light in seedling stadium and ramp that up over time.

For light, in addition, you would want to have a daily schedule to switch it on and off or even dim it softly. The amplitude would increase during the growing cycle but the daily schedule would still be needed.

Now you would set up

- a Time/Date method for let's say 3 month linearly increasing from 30% (seedlings) to 100% (harvest). f_year(t)

- a Daily (Time/Date) method that simulates sunrise and sunset between 0% and 100%. f_day(t)

- a Constant or Parameter Variable base method (as mentioned above) that returns 50%, as you know your lights are way to bright for lettuce. f_constant(t)

Next, you create a fourth method, a cascaded/combined one and assign all three methods to it.

This would now just do

f_light(t) = f_year(t) * f_day(t) * f_constant(t)

Now your daily light integral and peak light emissions would start softly at 30% (seedlings) * 100% (noon) * 50% (constant) = 15% for the seedlings and go up to maximum acceptable intensity at 100% (harvest) * 100% (noon) * 50% (constant)

Daylight Method

One final thing I did in my own proof of concept was daylight simulation. I configured the curves for a mid-summer day and scaled this day according to the real sunset sunrise times.

f_light( scale(t, sunrise, sunset) )

This then ends up in a flow like the one attached. I am aware that this is an extreme scenario, but who knows what happens. :-)

That's great. I'd gladly accept a PR. I'll then play around with implementing a Python code method. I haven't tested anything, but it all looks very neat and organized.

Please find my annotated pull request here: https://github.com/kizniche/Mycodo/pull/938

Glad to hear you're interested in playing aroung with it. I've not tried to add a new method yet, but the backend should find it automatically if you create a class "xyzMethod" (must end with Method, it will find it for method_type "xyz"). For now they are all in method.py, maybe we could break this up into several files in future.

If you create a method, you still need to announce it to the frontend, though.

Also, I'm not really sure about these very helpful "custom options" you implemented elsewhere. If we could add this to AbstractMethod, one could even create new methods without dealing with frontend changes - or allow user methods to be defined in each installation as with the inputs/outputs. 

That is fairly optional and for sure would be something that needs your expertise as the code author.

Adding methods into the frontend is still a bit painful, but finally I have manage to add the cascade method concept.

This is how it works:

- Create a new method of type "Method Cascade"

- In the details, add any number of existing methods. Even other cascade methods if you're cracking up. But don't worry, the implementation will prevent any recursive loops and put an error into the log.

- Use the cascade method in any function with debug enabled and watch the log:

2021-02-19 23:36:56,149 - DEBUG - mycodo.controllers.controller_trigger_6f600437 - [Method] Start: 2021-01-01 00:00:00 End: 2021-04-01 00:00:00
2021-02-19 23:36:56,155 - DEBUG - mycodo.controllers.controller_trigger_6f600437 - [Method] Start: 50.0 End: 100.0
2021-02-19 23:36:56,157 - DEBUG - mycodo.controllers.controller_trigger_6f600437 - [Method] Total: 7776000.0 Part total: 4318615.90673 (0.5553775600218621%)
2021-02-19 23:36:56,157 - DEBUG - mycodo.controllers.controller_trigger_6f600437 - [Method] New Setpoint: 77.7688780010931
2021-02-19 23:36:56,157 - DEBUG - mycodo.controllers.controller_trigger_6f600437 - Linked method: 6bf92a42-5238-4286-9d2c-adffaad65cf7 Year Intensity returned 77.7688780010931, False; current product is 77.7688780010931, False
2021-02-19 23:36:56,258 - DEBUG - mycodo.controllers.controller_trigger_6f600437 - [Method] Start: 19:00:00 End: 23:59:59
2021-02-19 23:36:56,258 - DEBUG - mycodo.controllers.controller_trigger_6f600437 - [Method] Start: 0.0 End: 0.0
2021-02-19 23:36:56,258 - DEBUG - mycodo.controllers.controller_trigger_6f600437 - [Method] Total: 17999.0 Part total: 16615.0 (0.9231068392688483%)
2021-02-19 23:36:56,259 - DEBUG - mycodo.controllers.controller_trigger_6f600437 - [Method] New Setpoint: 0.0
2021-02-19 23:36:56,259 - DEBUG - mycodo.controllers.controller_trigger_6f600437 - Linked method: bf056e2b-f87d-4e3e-a9ce-bae53b11be01 Daylight Time returned 0.0, False; current product is 0.0, False
2021-02-19 23:36:56,292 - DEBUG - mycodo.controllers.controller_trigger_6f600437 - Set output duty cycle to 0.0

Edit: Just found an issue if one combines in a duration method. As this normally stores it's start and end-time in the trigger to handle repeats, it doesn't work correctly as cascaded method. That's a pity because for this reason I invented it.

My demo use case: I've got a ventilator that should blow some air over the plants in regular intervals. But at night it should do that with reduced power. I combine a daily intensity curve with the duration that defindes the regular intervals. 

Will have to spend some more time with.

Okay, solved it. The problem was, that the restart of duration methods depended on the controller table. Therefore their execution was also tied to the controller.

Now the DurationMethod only depends on current time and start time to calculate its set_point.

My ventilator runs for some hours now using a combined duration method and a daily method for intensity change.

Let me know if you have any questions. 

https://github.com/kizniche/Mycodo/pull/939

You might still find my comments in the old PR 938 useful.

Oh, one thing. I had to add one column to the method_data table. I also added this to the model. How do database updates work in mycodo? I mean, that the new column is automatically created in existing databases.

CREATE TABLE method_data (

...

linked_method_id VARCHAR,

...

FOREIGN KEY (

linked_method_id

)

REFERENCES method (unique_id)

);

Thanks for the PR. I still need to go over the DB change you mention, but to explain the upgrade system:

To make a new database version file, use these commands:

cd Mycodo/databases

../env/bin/alembic revision -m "add x_option"

Which will create a new script in Mycodo/databases/alembic/versions. You can reference the other version scripts in that dir for the formatting to create a new column. There should be a few scripts that create a column with a foreign key and of how to then set an initial value. Be sure to change the alembic version at the top of the config.py to the latest script revision ID you just created. There is also a post_alembic script for performing more advanced DB manipulation if what you want to perform in the alembic upgrade script can't be performed for one reason or another. If you can't get the upgrade script figured out, I can make one later and show how I would structure it, in a reply here.

Also, to upgrade the DB after you have the script in place and edited, run:

cd Mycodo/databases

../env/bin/alembic upgrade head

I'd first make a backup of the database mycodo.db before performing the upgrade in case you want to vary the script and test the upgrade multiple times, you can just restore the DB and rerun the upgrade until you're happy with the result.

Thanks, excellent explanation. I've amended such a script to the pull request. Tested alembic upgrade, downgrade and re-upgrade.

As the alembic documentation states that foreign keys are a bit problematic in combination with sqlite, I relinquished the foreign key setting and created a normal independent column instead. As there is no update or delete cascade available, the foreign key constraint added little benefit anyway.

I added code to make sure entries of method and method_data tables stay in sync for this new field.

Should all be ready for your review and test now. Let me know if you have any questions or find some issues I can fix. Running my box with several classic as well asl several cascaded methods since end of last week without issues.

To explain what this database change is for, let's go through my example scenario.

Requirements:

- I've got a ventilator used for air recirculation.

- The ventilator should be turned on for 60 seconds each 5 minutes.

- The speed should be high at daylight time and low at nighttime

Solution

- Created a Daily Method that ramps up intensity from 20% to 100% in the morning and back to 20% in the evening

- Created a Duration Method that has 60 secs on and 240 secs off repeated indefinitely

- Created a Method Cascade and added the two other methods to it.

This ends up in the following data:

All three methods:

select unique_id, name, method_type from method;
unique_id                             name           method_type
------------------------------------  -------------  -----------
3dab2db5-f224-4362-9dda-c77e9f23352d  Schedule       Duration
a9722a6c-e81c-4805-aaaf-6c011351fab6  Intensity      Daily
cccf8a89-880f-438a-bb12-507b40591f11  Ventilator     Cascade

Method Data for the Cascade:

select unique_id, method_id, linked_method_id from method_data where method_id='cccf8a89-880f-438a-bb12-507b40591f11';
unique_id                             method_id                             linked_method_id
------------------------------------  ------------------------------------  ------------------------------------
ae2bc756-1d64-428b-a32b-1e1f3d7b110c  cccf8a89-880f-438a-bb12-507b40591f11  a9722a6c-e81c-4805-aaaf-6c011351fab6
5dfebd07-9419-45c6-8e14-83ff7f5e5a78  cccf8a89-880f-438a-bb12-507b40591f11  3dab2db5-f224-4362-9dda-c77e9f23352d

As you can see, the cascaded method cccf8a89-880f-438a-bb12-507b40591f11 has two method_data entry, each linked to one of the other methods.

What I added today is, that if you remove one of the linked methods, the link is also deleted.

This is of course a bit synthetic scenario that one could also have done similary by disabling and enabling the controller at times. But it was easiest to explain. More real scenario is to slowly increase daily light integral during the grow cycle.