| I&T Solution |
Virtual Construction Safety Assistant based on Multimodal Large Language Model embedded with Construction Safety Knowledge
(REF: S-1943) |
| Trial Project |
|
| Solution Feature |
- Bilingual multimodal large-small language model combining: (1) Low-Rank Adaptation (LoRA) co-adapters for multi-task and scene-adaptive learning, (2) multi-stage curriculum training framework based on Group Relative Policy Optimization (GRPO) algorithms with domain-tailored loss functions for reinforcement learning.
- Cloud-edge hybrid computing paradigm combining: (1) edge-deployed small models for low-level tasks including object identification and localization on images, (2) cloud-deployed large models for high-level safety compliance checking, with “reasoning token budget” and hierarchical reasoning algorithms for accelerated model response.
- Agentic retrieval-augmented generation (RAG) framework integrating: (1) chain-of-thoughts prompting for visual grounding of objects, (2) self-corrective query auto-generation for enhanced RAG driven by multiple agents (Retriever, Evaluator, Query Generator).
- Multi-stage self-supervised prompting framework integrating: (1) confidence scoring mechanism during first-stage reasoning for preliminary responses and feedback, (2) multi-image consistency cross-checking mechanism for differential comparison and adversarial self-verification for answer reinforcement, (3) semantic scene parsing and causal graph neural network for context-aware inference of root causes of safety hazards.
|
| Trial Application and Expected Outcome |
- Collaborate with Drainage Services Department (DSD) to identify an appropriate project site (e.g. Shek Wu Hui Sludge Treatment Works, conducted a similar project previously) and high-risk areas for configuring CCTV hardware and software to collect video data for technical development.
- Define the scope of safety rules to be monitored by reviewing safety regulations/standards (e.g. CIC work-at-height handbook), evaluation metrics and acceptance criteria with DSD.
- Deploy the solution as a software plugin to the CCTV system of DSD’s pilot site (e.g. for 6 months), and regularly enhance the system’s performance with accumulated data.
- Integrate with a customized user interface and instant alerting devices to offer automated video analysis capability on top of current smart site safety systems.
- Discuss with DSD the ownership of the deliverables (software licenses, intellectual properties), detailed paradigm of system maintenance and extended project scopes in the future.
|
| Additional Solution Information |
250609_Proposal_HKUST_JackCheng.pdf
|
| Info on I&T Solution Provider |
| Solution Provider | : | The Hong Kong University of Science and Technology | | Address | : | Room 3575 (Lift 27-28), Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong | | Contact Person | : | Prof. Jack Chin Pang CHENG |
| Position | : | Professor and Associate Head | | Tel | : | 51996561 | | Email | : |
cejcheng@ust.hk | | Webpage | : | https://ce.hkust.edu.hk/people/jack-chin-pang-cheng-zhengzhanpeng |
|
