Real-time gait detection and pose estimation are critical for safety monitoring and prevention of work-related musculoskeletal disorders for construction workers. We present a single wearable inertial measurement unit (IMU)-based gait detection and pose estimation for human walking on flat and sloped surfaces. The gait detection algorithm is built on a recurrent neural network-based method and its outcome is then used in the full-body pose estimation. The detection scheme also predicts the terrain slope information in real-time. The pose estimation is obtained through learned motion manifold in latent space with the Gaussian process dynamic model. Extensive experiments of different walking patterns and speeds on the level and sloped surfaces are conducted to validate and demonstrate the design. The proposed algorithm can detect gait activities with 96% accuracy, the estimated human pose errors are within 8.30 degs, and the detection latency is within 18.6 ms using only a single IMU attached to a human shank.