Enhance Gateway Network and Runtime Functionality

- Added a check in GatewayNetworkService::handleDaliRawFrame to ensure raw reporting is enabled for the gateway before processing the frame.
- Extended the JSON snapshot in GatewayNetworkService::gatewaySnapshotJson to include additional operation-related fields for each channel, providing more detailed status information.
- Updated GatewayRuntime::classifyCommandPriority to include new opcodes (0x66 and 0x67) in the control command classification, improving command handling.

Signed-off-by: Tony <tonylu@tony-cloud.com>

README.md

@@ -51,12 +51,14 @@ broadcast targets are never queried for refresh.
 Gateway feature opcode `0x06` keeps the Lua-compatible low-byte feature bits,
 advertises cache support with bit `0x40`, advertises gateway/channel name
 support with bit `0x80`, and advertises the native C++ gateway type with bit
-`0x0100`. Gateway opcode `0x05` reads and writes user-facing gateway identity:
+`0x0100`. Bit `0x0200` advertises the generic gateway operation protocol, and
-operation `0x00` reads the channel name, `0x01` writes the channel name,
+bit `0x0400` advertises explicit raw-report leases. Gateway opcode `0x05` reads
-`0x02` reads the physical gateway device name, and `0x03` writes the physical
+and writes user-facing gateway identity: operation `0x00` reads the channel
-gateway device name. Empty writes reset to the default name. Gateway opcode
+name, `0x01` writes the channel name, `0x02` reads the physical gateway device
-`0x39` returns cache summary and target snapshots so frontend clients can read
+name, and `0x03` writes the physical gateway device name. Empty writes reset to
-cached state without issuing live DALI queries on supported gateways.
+the default name. Gateway opcode `0x39` returns cache summary and target
+snapshots so frontend clients can read cached state without issuing live DALI
+queries on supported gateways.
 
 Gateway opcode `0x09` with address/data `0x00/0x00` is a chip-level channel-id
 report. It returns the enabled DALI channel ids so clients do not need to probe
@@ -73,6 +75,52 @@ channel number after the serial matches, so BLE/Wi-Fi configuration and DALI
 send/query commands can target a channel even when its variable channel id is
 unknown.
 
+## Gateway operation protocol
+
+Opcode `0x67` starts, aborts, and polls gateway-executed high-level DALI
+operations. Clients send one compact gateway command and the controller runs the
+required DTR setup, device-type select, query chain, retry, or address iteration
+on the gateway. The command frame is:
+
+- Start: `28 01 <gw> 67 01 <requestId> <operationIdLo> <operationIdHi> <payloadLenLo> <payloadLenHi> <TLV...> <checksum>`.
+- Abort: `28 01 <gw> 67 02 <requestId> 00 00 00 00 <checksum>`.
+- Status: `28 01 <gw> 67 03 <requestId> 00 00 00 00 <checksum>`.
+
+Multi-byte fields are little-endian. TLV payload fields use
+`{fieldId:u8, type:u8, len:u8, value...}`. Types are `0x01` u8, `0x02` u16,
+`0x03` u32, `0x04` signed i32, `0x05` bool, and `0x06` byte list. Dynamic
+snapshot metadata is returned as repeated byte-list entries using compact
+`key=value` payloads.
+
+Operation events are emitted as
+`22 67 <gw> <requestId> <operationIdLo> <operationIdHi> <event> <status> <progress> <target> <count> <checksum>`.
+Event values are accepted `0x00`, progress `0x01`, item result `0x02`,
+completed `0x03`, aborted `0x04`, and error `0x05`. Status values are ok
+`0x00`, busy `0x01`, invalid `0x02`, unsupported `0x03`, no response `0x04`,
+failed `0x05`, and aborted `0x06`.
+
+Large result payloads are emitted as chunks:
+`22 68 <gw> <requestId> <operationIdLo> <operationIdHi> <total> <index> <chunkLen> <TLV...> <checksum>`.
+Clients assemble chunks by gateway, request id, and operation id before
+processing result TLVs.
+
+The initial C++ executor accepts the shared `BridgeOperation` numeric ids for
+raw send/query, brightness/on/off/recall, color-temperature and DT8 RGB/XY/RGBW
+setters, DT1/DT4/DT5/DT6/DT8 snapshots, group masks, scene levels/maps, address
+settings, short-address range search, and short-address allocation/reset/stop.
+Unsupported ids return an `unsupported` operation error so old clients can fall
+back to their app-side workflow. Legacy opcodes such as `0x12`, `0x13`, `0x14`,
+`0x30`, `0x32`, `0x60`-`0x65`, and `0x39` remain available for compatibility.
+
+Opcode `0x66` controls passive raw-report leases. Command
+`28 01 <gw> 66 01 <enabled> <ttlLo> <ttlHi> <checksum>` enables or disables a
+volatile per-gateway lease; `ttl=0` disables. The response is
+`22 66 <gw> <status> <enabled> <ttlLo> <ttlHi> <checksum>`. Passive raw DALI
+notifications over UDP, BLE raw characteristics, ESP-NOW setup UART mirroring,
+and gateway notification opcodes `0x01` and `0x65` are suppressed unless the
+lease is active. Direct command/query responses still work without a raw-report
+lease.
+
 ## Current status
 
 The native rewrite now wires a shared `gateway_core` bootstrap component, a multi-channel `dali_domain` wrapper over `dali_cpp`, a local vendored `dali` hardware backend from the LuatOS ESP-IDF port with raw receive fan-out, an initial `gateway_runtime` service that provides persistent settings, device info, Lua-compatible command framing helpers, and Lua-style query command deduplication, plus a `gateway_controller` service that starts the gateway command task, dispatches core Lua gateway opcodes, and owns internal scene/group state. The gateway app also includes a `gateway_ble` NimBLE bridge that advertises a Lua-compatible GATT service and forwards `FFF3` framed notifications, incoming `FFF1`/`FFF2`/`FFF3` writes, and native raw DALI frame notifications into the matching raw channel, a `gateway_network` service that provides the native HTTP `/info`, `GET`/`POST /dali/cmd`, `/led/1`, `/led/0`, `/jq.js`, UDP control-plane router on port `2020`, W5500 SPI Ethernet with DHCP, Wi-Fi STA lifecycle, ESP-Touch smartconfig credential provisioning, the Lua-style `LAMMIN_Gateway` setup AP on `192.168.3.1`, ESP-NOW setup ingress for Lua-compatible `connReq`/`connAck`/`echo`/`cmd`/`data`/`uart` packets, native raw DALI frame forwarding back to connected setup peers, setup AP GPIO entry, and optional Wi-Fi credential reset GPIO handling, and an optional `gateway_485_control` bridge that claims UART0 for Lua-compatible framed command ingress plus `0x22` notification egress when the console is moved off UART0. Startup behavior is configured in `main/Kconfig.projbuild`: BLE and wired Ethernet are enabled by default, W5500 initialization and startup probe failures are ignored by default for boards without populated Ethernet hardware by fully disabling Ethernet for that boot, Wi-Fi STA, smartconfig, and ESP-NOW setup mode are disabled by default, the built-in USB Serial/JTAG interface stays in debug mode unless the optional USB setup bridge mode is selected, and the UART0 control bridge stays disabled unless the deployment explicitly repurposes UART0 away from the ESP-IDF console. Runtime settings and internal scene/group data are cached in RAM after load, skip unchanged flash writes, and batch Wi-Fi credential commits to reduce flash stalls on ESP32-S3 boards where flash and PSRAM share the SPI bus. The gateway app exposes per-channel PHY selection through `main/Kconfig.projbuild`; each channel can be disabled, bound to the native DALI GPIO HAL, or bound to a UART1/UART2 serial PHY. The checked-in `sdkconfig` is aligned with the app's custom 16 MB partition table so the Wi-Fi/BLE/network-enabled image fits the OTA app slots.

apps/gateway/.vscode/c_cpp_properties.json

@@ -0,0 +1,15 @@
+{
+    "configurations": [
+        {
+            "name": "Mac",
+            "includePath": [
+                "${workspaceFolder}/**"
+            ],
+            "defines": [],
+            "cStandard": "c17",
+            "cppStandard": "c++14",
+            "intelliSenseMode": "${default}"
+        }
+    ],
+    "version": 4
+}

apps/gateway/.vscode/settings.json

@@ -4,7 +4,7 @@
     "OPENOCD_SCRIPTS": "/Users/tonylu/.espressif/tools/openocd-esp32/v0.12.0-esp32-20240318/openocd-esp32/share/openocd/scripts",
     "ESP_ROM_ELF_DIR": "/Users/tonylu/.espressif/tools/esp-rom-elfs/20240305/"
   },
-  "clangd.path": "/Users/tonylu/.espressif/tools/esp-clang/esp-20.1.1_20250829/esp-clang/bin/clangd",
+  "clangd.path": "/Users/tonylu/.espressif/tools/esp-clang/esp-19.1.2_20250312/esp-clang/bin/clangd",
   "clangd.arguments": [
     "--background-index",
     "--query-driver=**",

components/gateway_ble/src/gateway_ble.cpp

@@ -405,6 +405,9 @@ void GatewayBleBridge::handleDaliRawFrame(const DaliRawFrame& frame) {
   if (!enabled_ || conn_handle_ == kInvalidConnectionHandle || frame.data.empty()) {
     return;
   }
+  if (!controller_.rawReportingEnabled(frame.gateway_id)) {
+    return;
+  }
   notifyCharacteristic(frame.channel_index, LegacyRawPayload(frame.data));
 }
 

components/gateway_controller/include/gateway_controller.hpp

@@ -52,6 +52,13 @@ struct GatewayChannelSnapshot {
   uint8_t group_mask_high{0};
   bool allocating{false};
   int last_alloc_addr{0};
+  bool operation_active{false};
+  uint8_t operation_request_id{0};
+  uint16_t operation_id{0};
+  uint8_t operation_status{0};
+  uint8_t operation_progress{0};
+  uint8_t operation_target{0};
+  uint8_t operation_count{0};
 };
 
 struct GatewayControllerSnapshot {
@@ -109,8 +116,16 @@ class GatewayController {
   bool bleEnabled() const;
   bool wifiEnabled() const;
   bool ipRouterEnabled() const;
+  bool rawReportingEnabled(uint8_t gateway_id) const;
   GatewayControllerSnapshot snapshot();
 
+  struct ParsedTlv {
+    uint8_t type{0};
+    std::vector<uint8_t> value;
+  };
+
+  using ParsedTlvMap = std::map<uint8_t, ParsedTlv>;
+
  private:
   struct ReconciliationJob {
     enum class Phase : uint8_t {
@@ -140,6 +155,30 @@ class GatewayController {
     uint8_t short_address{0};
   };
 
+  struct GatewayOperationRuntimeState {
+    bool active{false};
+    bool cancel_requested{false};
+    uint8_t request_id{0};
+    uint16_t operation_id{0};
+    uint8_t status{0};
+    uint8_t progress{0};
+    uint8_t target{0};
+    uint8_t count{0};
+  };
+
+  struct GatewayRawReportLease {
+    bool enabled{false};
+    TickType_t expires_at{0};
+  };
+
+  struct GatewayOperationTaskContext {
+    GatewayController* controller{nullptr};
+    uint8_t gateway_id{0};
+    uint8_t request_id{0};
+    uint16_t operation_id{0};
+    ParsedTlvMap fields;
+  };
+
   struct TransactionWaiter {
     uint8_t gateway_id{0};
     uint8_t opcode{0};
@@ -150,7 +189,9 @@ class GatewayController {
   };
 
   static void TaskEntry(void* arg);
+  static void OperationTaskEntry(void* arg);
   void taskLoop();
+  void runOperationTask(GatewayOperationTaskContext* context);
   void dispatchCommand(const std::vector<uint8_t>& command);
   void scheduleReconciliation(uint8_t gateway_id,
                               std::optional<GatewayCacheDaliTarget> target = std::nullopt);
@@ -171,6 +212,13 @@ class GatewayController {
   void refreshRuntimeGatewayNames();
   void publishPayload(uint8_t gateway_id, const std::vector<uint8_t>& payload);
   void publishFrame(const std::vector<uint8_t>& frame);
+  void publishRawReportLeaseResponse(uint8_t gateway_id, uint8_t status);
+  void publishOperationEvent(uint8_t gateway_id, uint8_t request_id, uint16_t operation_id,
+                             uint8_t event, uint8_t status, uint8_t progress,
+                             uint8_t target, uint8_t count);
+  void publishOperationResultChunks(uint8_t gateway_id, uint8_t request_id,
+                                    uint16_t operation_id,
+                                    const std::vector<uint8_t>& tlv_payload);
   bool transactionFrameMatches(const TransactionWaiter& waiter,
                                const std::vector<uint8_t>& frame) const;
   void captureTransactionFrame(const std::vector<uint8_t>& frame);
@@ -219,6 +267,16 @@ class GatewayController {
   bool gatewaySerialMatches(const std::vector<uint8_t>& command) const;
   void handleGatewayIdentityCommand(uint8_t gateway_id, uint8_t op);
   void handleAllocationCommand(uint8_t gateway_id, const std::vector<uint8_t>& command);
+  void handleRawReportLeaseCommand(uint8_t gateway_id, const std::vector<uint8_t>& command);
+  void handleGatewayOperationCommand(uint8_t gateway_id, const std::vector<uint8_t>& command);
+  bool operationActive(uint8_t gateway_id) const;
+  bool operationCancelRequested(uint8_t gateway_id, uint8_t request_id) const;
+  bool startOperation(uint8_t gateway_id, uint8_t request_id, uint16_t operation_id,
+                      ParsedTlvMap fields);
+  void abortOperation(uint8_t gateway_id, uint8_t request_id);
+  void finishOperation(uint8_t gateway_id, uint8_t request_id, uint8_t status,
+                       uint8_t progress, uint8_t target, uint8_t count);
+  ParsedTlvMap parseOperationTlvs(const uint8_t* data, size_t len, bool* ok) const;
   void handleInternalSceneCommand(uint8_t gateway_id, const std::vector<uint8_t>& command);
   void handleInternalGroupCommand(uint8_t gateway_id, const std::vector<uint8_t>& command);
   void handleGatewayCacheCommand(uint8_t gateway_id, const std::vector<uint8_t>& command);
@@ -234,6 +292,7 @@ class GatewayController {
   TaskHandle_t task_handle_{nullptr};
   SemaphoreHandle_t maintenance_lock_{nullptr};
   SemaphoreHandle_t transaction_lock_{nullptr};
+  SemaphoreHandle_t operation_lock_{nullptr};
   std::vector<NotificationSink> notification_sinks_;
   std::vector<BleStateSink> ble_state_sinks_;
   std::vector<WifiStateSink> wifi_state_sinks_;
@@ -242,6 +301,8 @@ class GatewayController {
   std::vector<TransactionWaiter*> transaction_waiters_;
   std::map<uint8_t, ReconciliationJob> reconciliation_jobs_;
   std::map<uint8_t, CacheRefreshJob> cache_refresh_jobs_;
+  std::map<uint8_t, GatewayOperationRuntimeState> operation_states_;
+  std::map<uint8_t, GatewayRawReportLease> raw_report_leases_;
   std::atomic<int> maintenance_activity_gateway_{-1};
   bool setup_mode_{false};
   bool wireless_setup_mode_{false};

components/gateway_network/src/gateway_network.cpp

@@ -1298,6 +1298,9 @@ void GatewayNetworkService::handleDaliRawFrame(const DaliRawFrame& frame) {
   if (!espnow_started_ || !espnow_connected_ || frame.data.empty()) {
     return;
   }
+  if (!controller_.rawReportingEnabled(frame.gateway_id)) {
+    return;
+  }
 
   cJSON* payload = cJSON_CreateObject();
   if (payload == nullptr) {
@@ -1606,6 +1609,13 @@ std::string GatewayNetworkService::gatewaySnapshotJson() {
       cJSON_AddNumberToObject(item, "groupMaskHigh", channel.group_mask_high);
       cJSON_AddBoolToObject(item, "isAllocAddr", channel.allocating);
       cJSON_AddNumberToObject(item, "lastAllocAddr", channel.last_alloc_addr);
+      cJSON_AddBoolToObject(item, "operationActive", channel.operation_active);
+      cJSON_AddNumberToObject(item, "operationRequestId", channel.operation_request_id);
+      cJSON_AddNumberToObject(item, "operationId", channel.operation_id);
+      cJSON_AddNumberToObject(item, "operationStatus", channel.operation_status);
+      cJSON_AddNumberToObject(item, "operationProgress", channel.operation_progress);
+      cJSON_AddNumberToObject(item, "operationTarget", channel.operation_target);
+      cJSON_AddNumberToObject(item, "operationCount", channel.operation_count);
       cJSON_AddItemToArray(channels, item);
     }
     cJSON_AddItemToObject(root, "channels", channels);
@@ -1757,4 +1767,4 @@ esp_err_t GatewayNetworkService::HandleJqJsGet(httpd_req_t* req) {
   return httpd_resp_send_chunk(req, nullptr, 0);
 }
 
-}  // namespace gateway
+}  // namespace gateway

components/gateway_runtime/src/gateway_runtime.cpp

@@ -370,7 +370,8 @@ GatewayRuntime::CommandPriority GatewayRuntime::classifyCommandPriority(
   if (opcode == 0x00 || opcode == 0x01 || opcode == 0x03 || opcode == 0x04 || opcode == 0x07 ||
       opcode == 0x08 || opcode == 0x10 || opcode == 0x11 || opcode == 0x12 || opcode == 0x13 ||
       opcode == 0x0B || opcode == 0x17 || opcode == 0x18 || opcode == 0x37 || opcode == 0x38 ||
-      opcode == 0x60 || opcode == 0x61 || opcode == 0x62 || (opcode == 0x30 && addr == 0)) {
+      opcode == 0x60 || opcode == 0x61 || opcode == 0x62 || opcode == 0x66 || opcode == 0x67 ||
+      (opcode == 0x30 && addr == 0)) {
     return CommandPriority::kControl;
   }
   return CommandPriority::kNormal;