💩 🔧 Enable suspending data collection for certain subgroups

[shankari]

This implements a hack for e-mission/e-mission-docs#1104
The request in the issue was to implement the ability to suspend and resume
data collection.

As a first step for implementing this, we:

• add a new section to the opcode in the dynamic config that lists the
  suspended subgroups, and
• skip all calls to usercache/put for users that are in suspended subgroup

This is a hack because the real fix is to stop data collection on the phone.

But even with this implementation:

• I have copied over the implementation of getSubgroupFromToken from the phone and manually ported it over to python. Instead, we should have all the implementations for parsing the opcode in e-mission-common and just use them here
• ideally, we would check for the subgroup being suspended in the putIntoCache method. But, by design, we only see the opcode in the auth module; the rest of the code works only with UUIDs. We should figure out a way to return the non-random parts of the opcode (program and subgroup) as a context so we can do additional validation downstream if needed. We can return this as a context variable, or store it into the profile and have the code check it from there.

@JGreenlee @TeachMeTW we should discuss if we want to continue with this hack on the server, or just go to the principled fix, in which case, most of these 💩 changes can be reverted.

[shankari]

Testing done:

After changing the default config

STUDY_CONFIG = os.getenv('STUDY_CONFIG', "dev-emulator-study")

and changing the dev-emulator-study config to have the new-style opcode with
a suspended subgroup

    "opcode": {
        "autogen": true,
        "subgroups": [
          "test",
          "default"
        ],
        "suspended_subgroups": ["default"]
    },

The suspended subgroup skipped all put messages

2025-02-02 18:57:35,723:DEBUG:6363951104:START POST /usercache/put
2025-02-02 18:57:35,724:DEBUG:6363951104:Called userCache.put
2025-02-02 18:57:35,742:DEBUG:6363951104:subgroup default found in list ['test', 'default']
2025-02-02 18:57:35,743:INFO:6363951104:Received put message for subgroup default in suspended
_subgroups=['default'], returning uuid = None
2025-02-02 18:57:35,743:DEBUG:6363951104:retUUID = None
2025-02-02 18:57:35,743:DEBUG:6380777472:START POST /usercache/get
2025-02-02 18:57:35,743:DEBUG:6363951104:END POST /usercache/put  0.023092985153198242

The non-suspended subgroup used the default implementation of put

2025-02-02 19:15:42,723:DEBUG:13035925504:START POST /usercache/put
2025-02-02 19:15:42,726:DEBUG:13035925504:Called userCache.put
2025-02-02 19:15:42,750:DEBUG:13035925504:subgroup test found in list ['test', 'default']
2025-02-02 19:15:42,759:DEBUG:13035925504:Updated result for user = feb70456-abf4-444b-8848-1515fc3470cf, key = stats/client_time, write_ts = 1738551626.711804 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('67a034de46756f8244ede842'), 'ok': 1.0, 'updatedExisting': False}
2025-02-02 19:15:42,761:DEBUG:13035925504:Updated result for user = feb70456-abf4-444b-8848-1515fc3470cf, key = stats/client_time, write_ts = 1738551626.71355 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('67a034de46756f8244ede844'), 'ok': 1.0, 'updatedExisting': False}

After modifying the config to remove suspended_subgroups (default case)

    "opcode": {
        "autogen": true,
        "subgroups": [
          "test",
          "default"
        ],
    },

2025-02-02 21:08:26,082:DEBUG:13052751872:START POST /usercache/put
2025-02-02 21:08:26,082:DEBUG:13052751872:Called userCache.put
2025-02-02 21:08:26,088:DEBUG:13052751872:subgroup default found in list ['test', 'default']
2025-02-02 21:08:26,088:DEBUG:13052751872:methodName = skip, returning <class 'emission.net.auth.skip.SkipMethod'>
2025-02-02 21:08:26,088:DEBUG:13052751872:Using the skip method to verify id token nrelop_dev-emulator-study_default_123 of length 37

2025-02-02 21:08:26,110:DEBUG:13052751872:Updated result for user = eb4a7aae-f2d4-4480-ba85-c568a45591b5, key = background/location, write_ts = 1738559116.0185199 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('67a04f4a46756f8244ee3a88'), 'ok': 1.0, 'updatedExisting': False}
2025-02-02 21:08:26,112:DEBUG:13052751872:Updated result for user = eb4a7aae-f2d4-4480-ba85-c568a45591b5, key = background/filtered_location, write_ts = 1738559116.020091 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('67a04f4a46756f8244ee3a8a'), 'ok': 1.0, 'updatedExisting': False}
...
2025-02-02 21:08:26,930:DEBUG:13052751872:Updated result for user = eb4a7aae-f2d4-4480-ba85-c568a45591b5, key = stats/client_nav_event, write_ts = 1738559306.0461679 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('67a04f4a46756f8244ee3da8'), 'ok': 1.0, 'updatedExisting': False}
2025-02-02 21:08:26,930:DEBUG:13052751872:END POST /usercache/put eb4a7aae-f2d4-4480-ba85-c568a45591b5 0.849830150604248

[shankari]

@mattwigway the hack has been merged now. It will be on staging for a couple of days, and then we will move it to production, likely by mid-week. After it is in production, you can choose to edit the config and suspend the subgroup at any time.

[mattwigway]

Great, thanks so much!

…

On Mon, Feb 3, 2025, 1:54 AM K. Shankari ***@***.***> wrote: Merged #1016 <#1016> into master. — Reply to this email directly, view it on GitHub <#1016 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEKNLVXKXFRBEXBF4DUTPL2N4HDFAVCNFSM6AAAAABWLH7FNOVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJWGE2TMOBXGUZDSMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[shankari]

We tested this on staging by:

• adding a new temp subgroup
• configuring two test phones to use the temp subgroup
• collecting data for a day (trips were uploaded and processed)
• suspending the temp subgroup and restarting the server to pick up the change
• collecting data for a day (upload was ignored)

2025-02-07 04:25:19,567:INFO:139885876868672:Received put message for subgroup temp in suspended_subgroups=['temp'], returning uuid = None
2025-02-07 04:27:43,377:INFO:139885834909248:Received put message for subgroup temp in suspended_subgroups=['temp'], returning uuid = None
2025-02-07 06:00:01,237:INFO:139886261958208:Received put message for subgroup temp in suspended_subgroups=['temp'], returning uuid = None
2025-02-07 06:55:25,869:INFO:139885824419392:Received put message for subgroup temp in suspended_subgroups=['temp'], returning uuid = None

• BUT argh! the change returns a 403 error to the phone, so the data keeps accumulating there.
• I need to change the server to return a 200; otherwise, this is just the same as shutting down the server.

13817,1738898733.391,2025-02-06T19:25:33.391000-08:00,BuiltinUserCache : Result count = 278
13833,1738898733.429,2025-02-06T19:25:33.429000-08:00,"ServerSyncAdapter : IO Error java.net.UnknownHostException: Unable to resolve host ""openpath-stage.nrel.gov"": No address associated with hostname while posting converted trips to JSON"
13835,1738898733.431,2025-02-06T19:25:33.431000-08:00,BuiltinUserCache : Added value for key stats/client_error at time 1.738898733431E9
13836,1738898733.434,2025-02-06T19:25:33.434000-08:00,"ServerSyncAdapter : Push complete, now pulling"

14449,1738902319.436,2025-02-06T20:25:19.436000-08:00,BuiltinUserCache : Result count = 278
14454,1738902319.736,2025-02-06T20:25:19.736000-08:00,CommunicationHelper : Got response org.apache.http.message.BasicHttpResponse@8ce8b76 with statusHTTP/1.1 403 Forbidden
14455,1738902319.755,2025-02-06T20:25:19.755000-08:00,ServerSyncAdapter : IO Error java.io.IOException while posting converted trips to JSON
14457,1738902319.759,2025-02-06T20:25:19.759000-08:00,"ServerSyncAdapter : Push complete, now pulling"

14768,1738902463.316,2025-02-06T20:27:43.316000-08:00,BuiltinUserCache : Result count = 356
14786,1738902463.551,2025-02-06T20:27:43.551000-08:00,CommunicationHelper : Got response org.apache.http.message.BasicHttpResponse@7c4d0d4 with statusHTTP/1.1 403 Forbidden
14787,1738902463.569,2025-02-06T20:27:43.569000-08:00,ServerSyncAdapter : IO Error java.io.IOException while posting converted trips to JSON
14789,1738902463.572,2025-02-06T20:27:43.572000-08:00,"ServerSyncAdapter : Push complete, now pulling"


14832,1738908000.968,2025-02-06T22:00:00.968000-08:00,BuiltinUserCache : Result count = 356
14837,1738908001.459,2025-02-06T22:00:01.459000-08:00,CommunicationHelper : Got response org.apache.http.message.BasicHttpResponse@193ea46 with statusHTTP/1.1 403 Forbidden
14838,1738908001.462,2025-02-06T22:00:01.462000-08:00,ServerSyncAdapter : IO Error java.io.IOException while posting converted trips to JSON
14840,1738908001.464,2025-02-06T22:00:01.464000-08:00,"ServerSyncAdapter : Push complete, now pulling"

[shankari]

The reason we are seeing the 403 error is because returning a None UUID errors out with a 403 even before the check in the usercache/put

        retUUID = enaa.getUUID(dynamic_config, request, auth_method, inHeader)
        logging.debug("retUUID = %s" % retUUID)
        if retUUID is None:
           raise HTTPError(403, "token is valid, but no account found for user")
        return retUUID

There are other ways to fix this (e.g. if the call is usercache/put, then don't return the 403), but instead of building a hack on top of a hack, I'm going to return the auth context instead and put the check into usercache/put as it should be.

[shankari]

Fixed the 403 error in #1018 and rebased to remove the changes here since they were 💩 anyway